Tracking Object and Session Changes with Events

Tracking Object and Session Changes with Events

SQLAlchemy features an extensive Event Listeningsystem used throughout the Core and ORM. Within the ORM, there are awide variety of event listener hooks, which are documented at an APIlevel at ORM Events. This collection of events hasgrown over the years to include lots of very useful new events as wellas some older events that aren’t as relevant as they once were. Thissection will attempt to introduce the major event hooks and when theymight be used.

Persistence Events

Probably the most widely used series of events are the “persistence” events,which correspond to the flush process.The flush is where all the decisions are made about pending changes toobjects and are then emitted out to the database in the form of INSERT,UPDATE, and DELETE statements.

before_flush()

The SessionEvents.before_flush() hook is by far the most generallyuseful event to use when an application wants to ensure thatadditional persistence changes to the database are made when a flush proceeds.Use SessionEvents.before_flush() in order to operateupon objects to validate their state as well as to compose additional objectsand references before they are persisted. Within this event,it is safe to manipulate the Session’s state, that is, new objectscan be attached to it, objects can be deleted, and individual attributeson objects can be changed freely, and these changes will be pulled intothe flush process when the event hook completes.

The typical SessionEvents.before_flush() hook will be tasked withscanning the collections Session.new, Session.dirty andSession.deleted in order to look for objectswhere something will be happening.

For illustrations of SessionEvents.before_flush(), seeexamples such as Versioning with a History Table andVersioning using Temporal Rows.

after_flush()

The SessionEvents.after_flush() hook is called after the SQL has beenemitted for a flush process, but before the state of the objects thatwere flushed has been altered. That is, you can still inspectthe Session.new, Session.dirty andSession.deleted collections to see what was just flushed, andyou can also use history tracking features like the ones providedby AttributeState to see what changes were just persisted.In the SessionEvents.after_flush() event, additional SQL can be emittedto the database based on what’s observed to have changed.

after_flush_postexec()

SessionEvents.after_flush_postexec() is called soon afterSessionEvents.after_flush(), but is invoked after the state ofthe objects has been modified to account for the flush that just took place.The Session.new, Session.dirty andSession.deleted collections are normally completely empty here.Use SessionEvents.after_flush_postexec() to inspect the identity mapfor finalized objects and possibly emit additional SQL. In this hook,there is the ability to make new changes on objects, which means theSession will again go into a “dirty” state; the mechanics of theSession here will cause it to flush again if new changesare detected in this hook if the flush were invoked in the context ofSession.commit(); otherwise, the pending changes will be bundledas part of the next normal flush. When the hook detects new changes withina Session.commit(), a counter ensures that an endless loop in thisregard is stopped after 100 iterations, in the case that anSessionEvents.after_flush_postexec()hook continually adds new state to be flushed each time it is called.

Mapper-level Events

In addition to the flush-level hooks, there is also a suite of hooksthat are more fine-grained, in that they are called on a per-objectbasis and are broken out based on INSERT, UPDATE or DELETE. Theseare the mapper persistence hooks, and they too are very popular,however these events need to be approached more cautiously, as theyproceed within the context of the flush process that is alreadyongoing; many operations are not safe to proceed here.

The events are:

Each event is passed the Mapper,the mapped object itself, and the Connection which is beingused to emit an INSERT, UPDATE or DELETE statement. The appeal of theseevents is clear, in that if an application wants to tie some activity towhen a specific type of object is persisted with an INSERT, the hook isvery specific; unlike the SessionEvents.before_flush() event,there’s no need to search through collections like Session.newin order to find targets. However, the flush plan whichrepresents the full list of every single INSERT, UPDATE, DELETE statementto be emitted has already been decided when these events are called,and no changes may be made at this stage. Therefore the only changes that areeven possible to the given objects are upon attributes local to theobject’s row. Any other change to the object or other objects willimpact the state of the Session, which will fail to functionproperly.

Operations that are not supported within these mapper-level persistenceevents include:

Session.add()
Session.delete()
Mapped collection append, add, remove, delete, discard, etc.
Mapped relationship attribute set/del events,i.e. someobject.related = someotherobject

The reason the Connection is passed is that it is encouraged thatsimple SQL operations take place here, directly on the Connection,such as incrementing counters or inserting extra rows within log tables.When dealing with the Connection, it is expected that Core-levelSQL operations will be used; e.g. those described in SQL Expression Language Tutorial.

There are also many per-object operations that don’t need to be handledwithin a flush event at all. The most common alternative is to simplyestablish additional state along with an object inside its init()method, such as creating additional objects that are to be associated withthe new object. Using validators as described in Simple Validators isanother approach; these functions can intercept changes to attributes andestablish additional state changes on the target object in response to theattribute change. With both of these approaches, the object is inthe correct state before it ever gets to the flush step.

Object Lifecycle Events

Another use case for events is to track the lifecycle of objects. Thisrefers to the states first introduced at Quickie Intro to Object States.

New in version 1.1: added a system of events that intercept all possiblestate transitions of an object within the Session.

All the states above can be tracked fully with events. Each eventrepresents a distinct state transition, meaning, the starting stateand the destination state are both part of what are tracked. With theexception of the initial transient event, all the events are in terms ofthe Session object or class, meaning they can be associated eitherwith a specific Session object:

from sqlalchemy import event
from sqlalchemy.orm import Session
 
session = Session()
 
@event.listens_for(session, 'transient_to_pending')
def object_is_pending(session, obj):
    print("new pending: %s" % obj)

Or with the Session class itself, as well as with a specificsessionmaker, which is likely the most useful form:

from sqlalchemy import event
from sqlalchemy.orm import sessionmaker
 
maker = sessionmaker()
 
@event.listens_for(maker, 'transient_to_pending')
def object_is_pending(session, obj):
    print("new pending: %s" % obj)

The listeners can of course be stacked on top of one function, as islikely to be common. For example, to track all objects that areentering the persistent state:

@event.listens_for(maker, "pending_to_persistent")@event.listens_for(maker, "deleted_to_persistent")@event.listens_for(maker, "detached_to_persistent")@event.listens_for(maker, "loaded_as_persistent")def detect_all_persistent(session, instance):    print("object is now persistent: %s" % instance)

Transient

All mapped objects when first constructed start out as transient.In this state, the object exists alone and doesn’t have an association withany Session. For this initial state, there’s no specific“transition” event since there is no Session, however if onewanted to intercept when any transient object is created, theInstanceEvents.init() method is probably the best event. Thisevent is applied to a specific class or superclass. For example, tointercept all new objects for a particular declarative base:

from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import event
 
Base = declarative_base()
 
@event.listens_for(Base, "init", propagate=True)
def intercept_init(instance, args, kwargs):
    print("new transient: %s" % instance)

Transient to Pending

The transient object becomes pending when it is first associatedwith a Session via the Session.add() or Session.add_all()method. An object may also become part of a Session as a resultof a “cascade” from a referencing object that wasexplicitly added. The transient to pending transition is detectable usingthe SessionEvents.transient_to_pending() event:

@event.listensfor(sessionmaker, "transient_to_pending")def intercept_transient_to_pending(session, object):    print("transient to pending: %s" % object_)

Pending to Persistent

The pending object becomes persistent when a flushproceeds and an INSERT statement takes place for the instance. The objectnow has an identity key. Track pending to persistent with theSessionEvents.pending_to_persistent() event:

@event.listensfor(sessionmaker, "pending_to_persistent")def intercept_pending_to_persistent(session, object):    print("pending to persistent: %s" % object_)

Pending to Transient

The pending object can revert back to transient if theSession.rollback() method is called before the pending objecthas been flushed, or if the Session.expunge() method is calledfor the object before it is flushed. Track pending to transient with theSessionEvents.pending_to_transient() event:

@event.listensfor(sessionmaker, "pending_to_transient")def intercept_pending_to_transient(session, object):    print("transient to pending: %s" % object_)

Loaded as Persistent

Objects can appear in the Session directly in the persistentstate when they are loaded from the database. Tracking this state transitionis synonymous with tracking objects as they are loaded, and is synonymouswith using the InstanceEvents.load() instance-level event. However, theSessionEvents.loaded_as_persistent() event is provided as asession-centric hook for intercepting objects as they enter the persistentstate via this particular avenue:

@event.listensfor(sessionmaker, "loaded_as_persistent")def intercept_loaded_as_persistent(session, object):    print("object loaded into persistent state: %s" % object_)

Persistent to Transient

The persistent object can revert to the transient state if theSession.rollback() method is called for a transaction where theobject was first added as pending. In the case of the ROLLBACK, theINSERT statement that made this object persistent is rolled back, andthe object is evicted from the Session to again become transient.Track objects that were reverted to transient frompersistent using the SessionEvents.persistent_to_transient()event hook:

@event.listensfor(sessionmaker, "persistent_to_transient")def intercept_persistent_to_transient(session, object):    print("persistent to transient: %s" % object_)

Persistent to Deleted

The persistent object enters the deleted state when an objectmarked for deletion is deleted from the database within the flushprocess. Note that this is not the same as when the Session.delete()method is called for a target object. The Session.delete()method only marks the object for deletion; the actual DELETE statementis not emitted until the flush proceeds. It is subsequent to the flushthat the “deleted” state is present for the target object.

Within the “deleted” state, the object is only marginally associatedwith the Session. It is not present in the identity mapnor is it present in the Session.deleted collection that refersto when it was pending for deletion.

From the “deleted” state, the object can go either to the detached statewhen the transaction is committed, or back to the persistent stateif the transaction is instead rolled back.

Track the persistent to deleted transition withSessionEvents.persistent_to_deleted():

@event.listensfor(sessionmaker, "persistent_to_deleted")def intercept_persistent_to_deleted(session, object):    print("object was DELETEd, is now in deleted state: %s" % object_)

Deleted to Detached

The deleted object becomes detached when the session’s transactionis committed. After the Session.commit() method is called, thedatabase transaction is final and the Session now fully discardsthe deleted object and removes all associations to it. Trackthe deleted to detached transition using SessionEvents.deleted_to_detached():

@event.listensfor(sessionmaker, "deleted_to_detached")def intercept_deleted_to_detached(session, object):    print("deleted to detached: %s" % object_)

Note

While the object is in the deleted state, the InstanceState.deletedattribute, accessible using inspect(object).deleted, returns True. Howeverwhen the object is detached, InstanceState.deleted will againreturn False. To detect that an object was deleted, regardless of whetheror not it is detached, use the InstanceState.was_deletedaccessor.

Persistent to Detached

The persistent object becomes detached when the object is de-associatedwith the Session, via the Session.expunge(),Session.expunge_all(), or Session.close() methods.

Note

An object may also become implicitly detached if its owningSession is dereferenced by the application and discarded due togarbage collection. In this case, no event is emitted.

Track objects as they move from persistent to detached using theSessionEvents.persistent_to_detached() event:

@event.listensfor(sessionmaker, "persistent_to_detached")def intercept_persistent_to_detached(session, object):    print("object became detached: %s" % object_)

Detached to Persistent

The detached object becomes persistent when it is re-associated with asession using the Session.add() or equivalent method. Trackobjects moving back to persistent from detached using theSessionEvents.detached_to_persistent() event:

@event.listensfor(sessionmaker, "detached_to_persistent")def intercept_detached_to_persistent(session, object):    print("object became persistent again: %s" % object_)

Deleted to Persistent

The deleted object can be reverted to the persistentstate when the transaction in which it was DELETEd was rolled backusing the Session.rollback() method. Track deleted objectsmoving back to the persistent state using theSessionEvents.deleted_to_persistent() event:

@event.listensfor(sessionmaker, "deleted_to_persistent")def intercept_deleted_to_persistent(session, object):    print("deleted to persistent: %s" % object_)

Transaction Events

Transaction events allow an application to be notified when transactionboundaries occur at the Session level as well as when theSession changes the transactional state on Connectionobjects.

SessionEvents.after_transaction_create(),SessionEvents.after_transaction_end() - these events track thelogical transaction scopes of the Session in a way that isnot specific to individual database connections. These events areintended to help with integration of transaction-tracking systems such aszope.sqlalchemy. Use theseevents when the application needs to align some external scope with thetransactional scope of the Session. These hooks mirrorthe “nested” transactional behavior of the Session, in that theytrack logical “subtransactions” as well as “nested” (e.g. SAVEPOINT)transactions.
SessionEvents.before_commit(), SessionEvents.after_commit(),SessionEvents.after_begin(),SessionEvents.after_rollback(), SessionEvents.after_soft_rollback() -These events allow tracking of transaction events from the perspectiveof database connections. SessionEvents.after_begin() in particularis a per-connection event; a Session that maintains more thanone connection will emit this event for each connection individuallyas those connections become used within the current transaction.The rollback and commit events then refer to when the DBAPI connectionsthemselves have received rollback or commit instructions directly.

Attribute Change Events

The attribute change events allow interception of when specific attributeson an object are modified. These events include AttributeEvents.set(),AttributeEvents.append(), and AttributeEvents.remove(). Theseevents are extremely useful, particularly for per-object validation operations;however, it is often much more convenient to use a “validator” hook, whichuses these hooks behind the scenes; see Simple Validators forbackground on this. The attribute events are also behind the mechanicsof backreferences. An example illustrating use of attribute eventsis in Attribute Instrumentation.