ORM Examples

The SQLAlchemy distribution includes a variety of code examples illustratinga select set of patterns, some typical and some not so typical. All arerunnable and can be found in the /examples directory of thedistribution. Descriptions and source code for all can be found here.

Additional SQLAlchemy examples, some user contributed, are available on thewiki at http://www.sqlalchemy.org/trac/wiki/UsageRecipes.

Mapping Recipes

Adjacency List

An example of a dictionary-of-dictionaries structure mapped usingan adjacency list model.

E.g.:

  1. node = TreeNode('rootnode')
  2. node.append('node1')
  3. node.append('node3')
  4. session.add(node)
  5. session.commit()
  6.  
  7. dump_tree(node)

Listing of files:

Associations

Examples illustrating the usage of the “association object” pattern,where an intermediary class mediates the relationship between twoclasses that are associated in a many-to-many pattern.

Listing of files:

proxied_association.py - Same example as basic_association, adding inusage of sqlalchemy.ext.associationproxy to make explicit referencesto OrderItem optional.

basic_association.py - Illustrate a many-to-many relationship between an“Order” and a collection of “Item” objects, associating a purchase pricewith each via an association object called “OrderItem”

dict_of_sets_with_default.py - An advanced association proxy example whichillustrates nesting of association proxies to produce multi-level Pythoncollections, in this case a dictionary with string keys and sets of integersas values, which conceal the underlying mapped classes.

Directed Graphs

An example of persistence for a directed graph structure. Thegraph is stored as a collection of edges, each referencing both a“lower” and an “upper” node in a table of nodes. Basic persistenceand querying for lower- and upper- neighbors are illustrated:

  1. n2 = Node(2)
  2. n5 = Node(5)
  3. n2.add_neighbor(n5)
  4. print n2.higher_neighbors()

Listing of files:

Dynamic Relations as Dictionaries

Illustrates how to place a dictionary-like facade on top of a“dynamic” relation, so that dictionary operations (assuming simplestring keys) can operate upon a large collection without loading thefull collection at once.

Listing of files:

Generic Associations

Illustrates various methods of associating multiple types ofparents with a particular child object.

The examples all use the declarative extension along withdeclarative mixins. Each one presents the identical usecase at the end - two classes, Customer and Supplier, bothsubclassing the HasAddresses mixin, which ensures that theparent class is provided with an addresses collectionwhich contains Address objects.

The discriminator_on_association.py and generic_fk.py scriptsare modernized versions of recipes presented in the 2007 blog postPolymorphic Associations with SQLAlchemy.

Listing of files:

table_per_association.py - Illustrates a mixin which provides a generic associationvia a individually generated association tables for each parent class.The associated objects themselves are persisted in a single tableshared among all parents.

table_per_related.py - Illustrates a generic association which persists associationobjects within individual tables, each one generated to persistthose objects on behalf of a particular parent class.

discriminator_on_association.py - Illustrates a mixin which provides a generic associationusing a single target table and a single association table,referred to by all parent tables. The association tablecontains a “discriminator” column which determines what type ofparent object associates to each particular row in the associationtable.

generic_fk.py - Illustrates a so-called “generic foreign key”, in a similar fashionto that of popular frameworks such as Django, ROR, etc. Thisapproach bypasses standard referential integritypractices, in that the “foreign key” column is not actuallyconstrained to refer to any particular table; instead,in-application logic is used to determine which table is referenced.

Large Collections

Large collection example.

Illustrates the options to use withrelationship() when the list of relatedobjects is very large, including:

  • “dynamic” relationships which query slices of data as accessed

  • how to use ON DELETE CASCADE in conjunction withpassive_deletes=True to greatly improve the performance ofrelated collection deletion.

Listing of files:

Materialized Paths

Illustrates the “materialized paths” pattern for hierarchical data using theSQLAlchemy ORM.

Listing of files:

materialized_paths.py - Illustrates the “materialized paths” pattern.

Nested Sets

Illustrates a rudimentary way to implement the “nested sets”pattern for hierarchical data using the SQLAlchemy ORM.

Listing of files:

nested_sets.py - Celko’s “Nested Sets” Tree Structure.

Performance

A performance profiling suite for a variety of SQLAlchemy use cases.

Each suite focuses on a specific use case with a particular performanceprofile and associated implications:

  • bulk inserts

  • individual inserts, with or without transactions

  • fetching large numbers of rows

  • running lots of short queries

All suites include a variety of use patterns illustrating both Coreand ORM use, and are generally sorted in order of performance from worstto greatest, inversely based on amount of functionality provided by SQLAlchemy,greatest to least (these two things generally correspond perfectly).

A command line tool is presented at the package level which allowsindividual suites to be run:

  1. $ python -m examples.performance --help
  2. usage: python -m examples.performance [-h] [--test TEST] [--dburl DBURL]
  3.                                       [--num NUM] [--profile] [--dump]
  4.                                       [--runsnake] [--echo]
  5.  
  6.                                       {bulk_inserts,large_resultsets,single_inserts}
  7.  
  8. positional arguments:
  9.   {bulk_inserts,large_resultsets,single_inserts}
  10.                         suite to run
  11.  
  12. optional arguments:
  13.   -h, --help            show this help message and exit
  14.   --test TEST           run specific test name
  15.   --dburl DBURL         database URL, default sqlite:///profile.db
  16.   --num NUM             Number of iterations/items/etc for tests;
  17.                         default is module-specific
  18.   --profile             run profiling and dump call counts
  19.   --dump                dump full call profile (implies --profile)
  20.   --runsnake            invoke runsnakerun (implies --profile)
  21.   --echo                Echo SQL output

An example run looks like:

  1. $ python -m examples.performance bulk_inserts

Or with options:

  1. $ python -m examples.performance bulk_inserts \
  2.     --dburl mysql+mysqldb://scott:tiger@localhost/test \
  3.     --profile --num 1000

See also

How can I profile a SQLAlchemy powered application?

File Listing

Listing of files:

bulk_updates.py - This series of tests illustrates different ways to UPDATE a large numberof rows in bulk.

large_resultsets.py - In this series of tests, we are looking at time to load a large numberof very small and simple rows.

bulk_inserts.py - This series of tests illustrates different ways to INSERT a large numberof rows in bulk.

short_selects.py - This series of tests illustrates different ways to SELECT a singlerecord by primary key

single_inserts.py - In this series of tests, we’re looking at a method that inserts a rowwithin a distinct transaction, and afterwards returns to essentially a“closed” state. This would be analogous to an API call that starts upa database connection, inserts the row, commits and closes.

main.py - Allows the examples/performance package to be run as a script.

Running all tests with time

This is the default form of run:

  1. $ python -m examples.performance single_inserts
  2. Tests to run: test_orm_commit, test_bulk_save,
  3.               test_bulk_insert_dictionaries, test_core,
  4.               test_core_query_caching, test_dbapi_raw_w_connect,
  5.               test_dbapi_raw_w_pool
  6.  
  7. test_orm_commit : Individual INSERT/COMMIT pairs via the
  8.     ORM (10000 iterations); total time 13.690218 sec
  9. test_bulk_save : Individual INSERT/COMMIT pairs using
  10.     the "bulk" API  (10000 iterations); total time 11.290371 sec
  11. test_bulk_insert_dictionaries : Individual INSERT/COMMIT pairs using
  12.     the "bulk" API with dictionaries (10000 iterations);
  13.     total time 10.814626 sec
  14. test_core : Individual INSERT/COMMIT pairs using Core.
  15.     (10000 iterations); total time 9.665620 sec
  16. test_core_query_caching : Individual INSERT/COMMIT pairs using Core
  17.     with query caching (10000 iterations); total time 9.209010 sec
  18. test_dbapi_raw_w_connect : Individual INSERT/COMMIT pairs w/ DBAPI +
  19.     connection each time (10000 iterations); total time 9.551103 sec
  20. test_dbapi_raw_w_pool : Individual INSERT/COMMIT pairs w/ DBAPI +
  21.     connection pool (10000 iterations); total time 8.001813 sec

Dumping Profiles for Individual Tests

A Python profile output can be dumped for all tests, or more commonlyindividual tests:

  1. $ python -m examples.performance single_inserts --test test_core --num 1000 --dump
  2. Tests to run: test_core
  3. test_core : Individual INSERT/COMMIT pairs using Core. (1000 iterations); total fn calls 186109
  4.          186109 function calls (186102 primitive calls) in 1.089 seconds
  5.  
  6.    Ordered by: internal time, call count
  7.  
  8.    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  9.      1000    0.634    0.001    0.634    0.001 {method 'commit' of 'sqlite3.Connection' objects}
  10.      1000    0.154    0.000    0.154    0.000 {method 'execute' of 'sqlite3.Cursor' objects}
  11.      1000    0.021    0.000    0.074    0.000 /Users/classic/dev/sqlalchemy/lib/sqlalchemy/sql/compiler.py:1950(_get_colparams)
  12.      1000    0.015    0.000    0.034    0.000 /Users/classic/dev/sqlalchemy/lib/sqlalchemy/engine/default.py:503(_init_compiled)
  13.         1    0.012    0.012    1.091    1.091 examples/performance/single_inserts.py:79(test_core)
  14.  
  15.     ...

Using RunSnake

This option requires the RunSnakecommand line tool be installed:

  1. $ python -m examples.performance single_inserts --test test_core --num 1000 --runsnake

A graphical RunSnake output will be displayed.

Writing your Own Suites

The profiler suite system is extensible, and can be applied to your own setof tests. This is a valuable technique to use in deciding upon the properapproach for some performance-critical set of routines. For example,if we wanted to profile the difference between several kinds of loading,we can create a file test_loads.py, with the following content:

  1. from examples.performance import Profiler
  2. from sqlalchemy import Integer, Column, create_engine, ForeignKey
  3. from sqlalchemy.orm import relationship, joinedload, subqueryload, Session
  4. from sqlalchemy.ext.declarative import declarative_base
  5.  
  6. Base = declarative_base()
  7. engine = None
  8. session = None
  9.  
  10.  
  11. class Parent(Base):
  12.     __tablename__ = 'parent'
  13.     id = Column(Integer, primary_key=True)
  14.     children = relationship("Child")
  15.  
  16.  
  17. class Child(Base):
  18.     __tablename__ = 'child'
  19.     id = Column(Integer, primary_key=True)
  20.     parent_id = Column(Integer, ForeignKey('parent.id'))
  21.  
  22.  
  23. # Init with name of file, default number of items
  24. Profiler.init("test_loads", 1000)
  25.  
  26.  
  27. @Profiler.setup_once
  28. def setup_once(dburl, echo, num):
  29.     "setup once.  create an engine, insert fixture data"
  30.     global engine
  31.     engine = create_engine(dburl, echo=echo)
  32.     Base.metadata.drop_all(engine)
  33.     Base.metadata.create_all(engine)
  34.     sess = Session(engine)
  35.     sess.add_all([
  36.         Parent(children=[Child() for j in range(100)])
  37.         for i in range(num)
  38.     ])
  39.     sess.commit()
  40.  
  41.  
  42. @Profiler.setup
  43. def setup(dburl, echo, num):
  44.     "setup per test.  create a new Session."
  45.     global session
  46.     session = Session(engine)
  47.     # pre-connect so this part isn't profiled (if we choose)
  48.     session.connection()
  49.  
  50.  
  51. @Profiler.profile
  52. def test_lazyload(n):
  53.     "load everything, no eager loading."
  54.  
  55.     for parent in session.query(Parent):
  56.         parent.children
  57.  
  58.  
  59. @Profiler.profile
  60. def test_joinedload(n):
  61.     "load everything, joined eager loading."
  62.  
  63.     for parent in session.query(Parent).options(joinedload("children")):
  64.         parent.children
  65.  
  66.  
  67. @Profiler.profile
  68. def test_subqueryload(n):
  69.     "load everything, subquery eager loading."
  70.  
  71.     for parent in session.query(Parent).options(subqueryload("children")):
  72.         parent.children
  73.  
  74. if __name__ == '__main__':
  75.     Profiler.main()

We can run our new script directly:

  1. $ python test_loads.py  --dburl postgresql+psycopg2://scott:tiger@localhost/test
  2. Running setup once...
  3. Tests to run: test_lazyload, test_joinedload, test_subqueryload
  4. test_lazyload : load everything, no eager loading. (1000 iterations); total time 11.971159 sec
  5. test_joinedload : load everything, joined eager loading. (1000 iterations); total time 2.754592 sec
  6. test_subqueryload : load everything, subquery eager loading. (1000 iterations); total time 2.977696 sec

As well as see RunSnake output for an individual test:

  1. $ python test_loads.py  --num 100 --runsnake --test test_joinedload

Relationship Join Conditions

Examples of various orm.relationship() configurations,which make use of the primaryjoin argument to compose special typesof join conditions.

Listing of files:

threeway.py - Illustrate a “three way join” - where a primary table joins to a remotetable via an association table, but then the primary table also needsto refer to some columns in the remote table directly.

cast.py - Illustrate a relationship() that joins two columns where thosecolumns are not of the same type, and a CAST must be used on the SQLside in order to match them.

Space Invaders

A Space Invaders game using SQLite as the state machine.

Originally developed in 2012. Adapted to work in Python 3.

Runs in a textual console using ASCII art.../_images/space_invaders.jpgTo run:

  1. python -m examples.space_invaders.space_invaders

While it runs, watch the SQL output in the log:

  1. tail -f space_invaders.log

enjoy!

Listing of files:

XML Persistence

Illustrates three strategies for persisting and querying XMLdocuments as represented by ElementTree in a relationaldatabase. The techniques do not apply any mappings to theElementTree objects directly, so are compatible with thenative cElementTree as well as lxml, and can be adapted tosuit any kind of DOM representation system. Querying alongxpath-like strings is illustrated as well.

E.g.:

  1. # parse an XML file and persist in the database
  2. doc = ElementTree.parse("test.xml")
  3. session.add(Document(file, doc))
  4. session.commit()
  5.  
  6. # locate documents with a certain path/attribute structure
  7. for document in find_document('/somefile/header/field2[@attr=foo]'):
  8.     # dump the XML
  9.     print document

Listing of files:

pickle_type.py - illustrates a quick and dirty way to persist an XML document expressed usingElementTree and pickle.

adjacency_list.py - Illustrates an explicit way to persist an XML document expressed usingElementTree.

optimized_al.py - Uses the same strategy as adjacency_list.py, but associates each DOM row with its owning document row, so that a full document of DOM nodes can be loaded using O(1) queries - the construction of the “hierarchy” is performed after the load in a non-recursive fashion and is more efficient.

Versioning Objects

Versioning with a History Table

Illustrates an extension which creates version tables for entities and storesrecords for each change. The given extensions generate an anonymous “history”class which represents historical versions of the target object.

Compare to the Versioning using Temporal Rows examples which write updatesas new rows in the same table, without using a separate history table.

Usage is illustrated via a unit test module test_versioning.py, which canbe run via py.test:

  1. # assume SQLAlchemy is installed where py.test is
  2.  
  3. cd examples/versioned_history
  4. py.test test_versioning.py

A fragment of example usage, using declarative:

  1. from history_meta import Versioned, versioned_session
  2.  
  3. Base = declarative_base()
  4.  
  5. class SomeClass(Versioned, Base):
  6.     __tablename__ = 'sometable'
  7.  
  8.     id = Column(Integer, primary_key=True)
  9.     name = Column(String(50))
  10.  
  11.     def __eq__(self, other):
  12.         assert type(other) is SomeClass and other.id == self.id
  13.  
  14. Session = sessionmaker(bind=engine)
  15. versioned_session(Session)
  16.  
  17. sess = Session()
  18. sc = SomeClass(name='sc1')
  19. sess.add(sc)
  20. sess.commit()
  21.  
  22. sc.name = 'sc1modified'
  23. sess.commit()
  24.  
  25. assert sc.version == 2
  26.  
  27. SomeClassHistory = SomeClass.__history_mapper__.class_
  28.  
  29. assert sess.query(SomeClassHistory).\
  30.             filter(SomeClassHistory.version == 1).\
  31.             all() \
  32.             == [SomeClassHistory(version=1, name='sc1')]

The Versioned mixin is designed to work with declarative. To usethe extension with classical mappers, the _history_mapper functioncan be applied:

  1. from history_meta import _history_mapper
  2.  
  3. m = mapper(SomeClass, sometable)
  4. _history_mapper(m)
  5.  
  6. SomeHistoryClass = SomeClass.__history_mapper__.class_

Listing of files:

test_versioning.py - Unit tests illustrating usage of the history_meta.pymodule functions.

history_meta.py - Versioned mixin class and other utilities.

Versioning using Temporal Rows

Several examples that illustrate the technique of intercepting changesthat would be first interpreted as an UPDATE on a row, and instead turningit into an INSERT of a new row, leaving the previous row intact asa historical version.

Compare to the Versioning with a History Table example which writes ahistory row to a separate history table.

Listing of files:

versioned_rows.py - Illustrates a method to intercept changes on objects, turningan UPDATE statement on a single row into an INSERT statement, so that a newrow is inserted with the new data, keeping the old row intact.

versioned_rows_w_versionid.py - Illustrates a method to intercept changes on objects, turningan UPDATE statement on a single row into an INSERT statement, so that a newrow is inserted with the new data, keeping the old row intact.

versioned_map.py - A variant of the versioned_rows example built around theconcept of a “vertical table” structure, like those illustrated inVertical Attribute Mapping examples.

versioned_update_old_row.py - Illustrates the same UPDATE into INSERT technique of versioned_rows.py,but also emits an UPDATE on the old row to affect a change in timestamp.Also includes a QueryEvents.before_compile() hook to limit queriesto only the most recent version.

Vertical Attribute Mapping

Illustrates “vertical table” mappings.

A “vertical table” refers to a technique where individual attributesof an object are stored as distinct rows in a table. The “verticaltable” technique is used to persist objects which can have a variedset of attributes, at the expense of simple query control and brevity.It is commonly found in content/document management systems in orderto represent user-created structures flexibly.

Two variants on the approach are given. In the second, each rowreferences a “datatype” which contains information about the type ofinformation stored in the attribute, such as integer, string, or date.

Example:

  1. shrew = Animal(u'shrew')
  2. shrew[u'cuteness'] = 5
  3. shrew[u'weasel-like'] = False
  4. shrew[u'poisonous'] = True
  5.  
  6. session.add(shrew)
  7. session.flush()
  8.  
  9. q = (session.query(Animal).
  10.      filter(Animal.facts.any(
  11.        and_(AnimalFact.key == u'weasel-like',
  12.             AnimalFact.value == True))))
  13. print 'weasel-like animals', q.all()

Listing of files:

dictlike-polymorphic.py - Mapping a polymorphic-valued vertical table as a dictionary.

dictlike.py - Mapping a vertical table as a dictionary.

Inheritance Mapping Recipes

Basic Inheritance Mappings

Working examples of single-table, joined-table, and concrete-tableinheritance as described in Mapping Class Inheritance Hierarchies.

Listing of files:

joined.py - Joined-table (table-per-subclass) inheritance example.

concrete.py - Concrete-table (table-per-class) inheritance example.

single.py - Single-table (table-per-hierarchy) inheritance example.

Special APIs

Attribute Instrumentation

Examples illustrating modifications to SQLAlchemy’s attribute managementsystem.

Listing of files:

listen_for_events.py - Illustrates how to attach events to all instrumented attributesand listen for change events.

active_column_defaults.py - Illustrates use of the AttributeEvents.init_scalar()event, in conjunction with Core column defaults to provideORM objects that automatically produce the default valuewhen an un-set attribute is accessed.

custom_management.py - Illustrates customized class instrumentation, usingthe sqlalchemy.ext.instrumentation extension package.

Horizontal Sharding

A basic example of using the SQLAlchemy Sharding API.Sharding refers to horizontally scaling data across multipledatabases.

The basic components of a “sharded” mapping are:

  • multiple databases, each assigned a ‘shard id’

  • a function which can return a single shard id, given an instanceto be saved; this is called “shard_chooser”

  • a function which can return a list of shard ids which apply to a particularinstance identifier; this is called “id_chooser”.If it returns all shard ids,all shards will be searched.

  • a function which can return a list of shard ids to try, given a particularQuery (“query_chooser”). If it returns all shard ids, all shards will bequeried and the results joined together.

In this example, four sqlite databases will store information about weatherdata on a database-per-continent basis. We provide example shard_chooser,id_chooser and query_chooser functions. The query_chooser illustratesinspection of the SQL expression element in order to attempt to determine asingle shard being requested.

The construction of generic sharding routines is an ambitious approachto the issue of organizing instances among multiple databases. For amore plain-spoken alternative, the “distinct entity” approachis a simple method of assigning objects to different tables (and potentiallydatabase nodes) in an explicit way - described on the wiki atEntityName.

Listing of files:

Extending the ORM

Dogpile Caching

Illustrates how to embeddogpile.cachefunctionality within the Query object, allowing full cache controlas well as the ability to pull “lazy loaded” attributes from long term cache.

In this demo, the following techniques are illustrated:

  • Using custom subclasses of Query

  • Basic technique of circumventing Query to pull from acustom cache source instead of the database.

  • Rudimental caching with dogpile.cache, using “regions” which allowglobal control over a fixed set of configurations.

  • Using custom MapperOption objects to configure options ona Query, including the ability to invoke the optionsdeep within an object graph when lazy loads occur.

E.g.:

  1. # query for Person objects, specifying cache
  2. q = Session.query(Person).options(FromCache("default"))
  3.  
  4. # specify that each Person's "addresses" collection comes from
  5. # cache too
  6. q = q.options(RelationshipCache(Person.addresses, "default"))
  7.  
  8. # query
  9. print q.all()

To run, both SQLAlchemy and dogpile.cache must beinstalled or on the current PYTHONPATH. The demo will create a localdirectory for datafiles, insert initial data, and run. Running thedemo a second time will utilize the cache files already present, andexactly one SQL statement against two tables will be emitted - thedisplayed result however will utilize dozens of lazyloads that allpull from cache.

The demo scripts themselves, in order of complexity, are run as Pythonmodules so that relative imports work:

  1. python -m examples.dogpile_caching.helloworld
  2.  
  3. python -m examples.dogpile_caching.relationship_caching
  4.  
  5. python -m examples.dogpile_caching.advanced
  6.  
  7. python -m examples.dogpile_caching.local_session_caching

Listing of files:

environment.py - Establish data / cache file paths, and configurations,bootstrap fixture data if necessary.

caching_query.py - Represent functions and classeswhich allow the usage of Dogpile caching with SQLAlchemy.Introduces a query option called FromCache.

model.py - The datamodel, which represents Person that has multipleAddress objects, each with PostalCode, City, Country.

fixture_data.py - Installs some sample data. Here we have a handful of postal codes fora few US/Canadian cities. Then, 100 Person records are installed, eachwith a randomly selected postal code.

helloworld.py - Illustrate how to load some data, and cache the results.

relationship_caching.py - Illustrates how to add cache options onrelationship endpoints, so that lazyloads load from cache.

advanced.py - Illustrate usage of Query combined with the FromCache option,including front-end loading, cache invalidation and collection caching.

local_session_caching.py - This example creates a new dogpile.cache backend that will persist data in adictionary which is local to the current session. remove() the session andthe cache is gone.

PostGIS Integration

A naive example illustrating techniques to helpembed PostGIS functionality.

This example was originally developed in the hopes that it would beextrapolated into a comprehensive PostGIS integration layer. We arepleased to announce that this has come to fruition as GeoAlchemy.

The example illustrates:

  • a DDL extension which allows CREATE/DROP to work inconjunction with AddGeometryColumn/DropGeometryColumn

  • a Geometry type, as well as a few subtypes, whichconvert result row values to a GIS-aware object,and also integrates with the DDL extension.

  • a GIS-aware object which stores a raw geometry valueand provides a factory for functions such as AsText().

  • an ORM comparator which can override standard columnmethods on mapped objects to produce GIS operators.

  • an attribute event listener that intercepts stringsand converts to GeomFromText().

  • a standalone operator example.

The implementation is limited to only public, well knownand simple to use extension points.

E.g.:

  1. print session.query(Road).filter(
  2.     Road.road_geom.intersects(r1.road_geom)).all()

Listing of files: