High Availability and PyMongo

PyMongo makes it easy to write highly available applications whetheryou use a single replica setor a large sharded cluster.

Connecting to a Replica Set

PyMongo makes working with replica sets easy. Here we’ll launch a newreplica set and show how to handle both initialization and normalconnections with PyMongo.

See also

The MongoDB documentation on

rs

Starting a Replica Set

The main replica set documentation contains extensive informationabout setting up a new replica set or migrating an existing MongoDBsetup, be sure to check that out. Here, we’ll just do the bare minimumto get a three node replica set setup locally.

Warning

Replica sets should always use multiple nodes inproduction - putting all set members on the same physical node isonly recommended for testing and development.

We start three mongod processes, each on a different port and witha different dbpath, but all using the same replica set name “foo”.

  1. $ mkdir -p /data/db0 /data/db1 /data/db2
  2. $ mongod --port 27017 --dbpath /data/db0 --replSet foo
  1. $ mongod --port 27018 --dbpath /data/db1 --replSet foo
  1. $ mongod --port 27019 --dbpath /data/db2 --replSet foo

Initializing the Set

At this point all of our nodes are up and running, but the set has yetto be initialized. Until the set is initialized no node will becomethe primary, and things are essentially “offline”.

To initialize the set we need to connect to a single node and run theinitiate command:

  1. >>> from pymongo import MongoClient
  2. >>> c = MongoClient('localhost', 27017)

Note

We could have connected to any of the other nodes instead,but only the node we initiate from is allowed to contain anyinitial data.

After connecting, we run the initiate command to get things started:

  1. >>> config = {'_id': 'foo', 'members': [
  2. ...     {'_id': 0, 'host': 'localhost:27017'},
  3. ...     {'_id': 1, 'host': 'localhost:27018'},
  4. ...     {'_id': 2, 'host': 'localhost:27019'}]}
  5. >>> c.admin.command("replSetInitiate", config)
  6. {'ok': 1.0, ...}

The three mongod servers we started earlier will now coordinateand come online as a replica set.

Connecting to a Replica Set

The initial connection as made above is a special case for anuninitialized replica set. Normally we’ll want to connectdifferently. A connection to a replica set can be made using theMongoClient() constructor, specifyingone or more members of the set, along with the replica set name. Any ofthe following connects to the replica set we just created:

  1. >>> MongoClient('localhost', replicaset='foo')
  2. MongoClient(host=['localhost:27017'], replicaset='foo', ...)
  3. >>> MongoClient('localhost:27018', replicaset='foo')
  4. MongoClient(['localhost:27018'], replicaset='foo', ...)
  5. >>> MongoClient('localhost', 27019, replicaset='foo')
  6. MongoClient(['localhost:27019'], replicaset='foo', ...)
  7. >>> MongoClient('mongodb://localhost:27017,localhost:27018/?replicaSet=foo')
  8. MongoClient(['localhost:27017', 'localhost:27018'], replicaset='foo', ...)

The addresses passed to MongoClient() are calledthe seeds. As long as at least one of the seeds is online, MongoClientdiscovers all the members in the replica set, and determines which is thecurrent primary and which are secondaries or arbiters. Each seed must be theaddress of a single mongod. Multihomed and round robin DNS addresses arenot supported.

The MongoClient constructor is non-blocking:the constructor returns immediately while the client connects to the replicaset using background threads. Note how, if you create a client and immediatelyprint the string representation of itsnodes attribute, the list may beempty initially. If you wait a moment, MongoClient discovers the whole replicaset:

  1. >>> from time import sleep
  2. >>> c = MongoClient(replicaset='foo'); print(c.nodes); sleep(0.1); print(c.nodes)
  3. frozenset([])
  4. frozenset([(u'localhost', 27019), (u'localhost', 27017), (u'localhost', 27018)])

You need not wait for replica set discovery in your application, however.If you need to do any operation with a MongoClient, such as afind() or aninsert_one(), the client waits to discovera suitable member before it attempts the operation.

Handling Failover

When a failover occurs, PyMongo will automatically attempt to find thenew primary node and perform subsequent operations on that node. Thiscan’t happen completely transparently, however. Here we’ll perform anexample failover to illustrate how everything behaves. First, we’llconnect to the replica set and perform a couple of basic operations:

  1. >>> db = MongoClient("localhost", replicaSet='foo').test
  2. >>> db.test.insert_one({"x": 1}).inserted_id
  3. ObjectId('...')
  4. >>> db.test.find_one()
  5. {u'x': 1, u'_id': ObjectId('...')}

By checking the host and port, we can see that we’re connected tolocalhost:27017, which is the current primary:

  1. >>> db.client.address
  2. ('localhost', 27017)

Now let’s bring down that node and see what happens when we run ourquery again:

  1. >>> db.test.find_one()
  2. Traceback (most recent call last):
  3. pymongo.errors.AutoReconnect: ...

We get an AutoReconnect exception. This meansthat the driver was not able to connect to the old primary (whichmakes sense, as we killed the server), but that it will attempt toautomatically reconnect on subsequent operations. When this exceptionis raised our application code needs to decide whether to retry theoperation or to simply continue, accepting the fact that the operationmight have failed.

On subsequent attempts to run the query we might continue to see thisexception. Eventually, however, the replica set will failover andelect a new primary (this should take no more than a couple of seconds ingeneral). At that point the driver will connect to the new primary andthe operation will succeed:

  1. >>> db.test.find_one()
  2. {u'x': 1, u'_id': ObjectId('...')}
  3. >>> db.client.address
  4. ('localhost', 27018)

Bring the former primary back up. It will rejoin the set as a secondary.Now we can move to the next section: distributing reads to secondaries.

Secondary Reads

By default an instance of MongoClient sends queries tothe primary member of the replica set. To use secondaries for querieswe have to change the read preference:

  1. >>> client = MongoClient(
  2. ...     'localhost:27017',
  3. ...     replicaSet='foo',
  4. ...     readPreference='secondaryPreferred')
  5. >>> client.read_preference
  6. SecondaryPreferred(tag_sets=None)

Now all queries will be sent to the secondary members of the set. If there areno secondary members the primary will be used as a fallback. If you havequeries you would prefer to never send to the primary you can specify thatusing the secondary read preference.

By default the read preference of a Database isinherited from its MongoClient, and the read preference of aCollection is inherited from its Database. To usea different read preference use theget_database() method, or theget_collection() method:

  1. >>> from pymongo import ReadPreference
  2. >>> client.read_preference
  3. SecondaryPreferred(tag_sets=None)
  4. >>> db = client.get_database('test', read_preference=ReadPreference.SECONDARY)
  5. >>> db.read_preference
  6. Secondary(tag_sets=None)
  7. >>> coll = db.get_collection('test', read_preference=ReadPreference.PRIMARY)
  8. >>> coll.read_preference
  9. Primary()

You can also change the read preference of an existingCollection with thewith_options() method:

  1. >>> coll2 = coll.with_options(read_preference=ReadPreference.NEAREST)
  2. >>> coll.read_preference
  3. Primary()
  4. >>> coll2.read_preference
  5. Nearest(tag_sets=None)

Note that since most database commands can only be sent to the primary of areplica set, the command() method does not obeythe Database’s read_preference, but you canpass an explicit read preference to the method:

  1. >>> db.command('dbstats', read_preference=ReadPreference.NEAREST)
  2. {...}

Reads are configured using three options: read preference, tag sets,and local threshold.

Read preference:

Read preference is configured using one of the classes fromread_preferences (Primary,PrimaryPreferred,Secondary,SecondaryPreferred, orNearest). For convenience, we also provideReadPreference with the followingattributes:

  • PRIMARY: Read from the primary. This is the default read preference,and provides the strongest consistency. If no primary is available, raiseAutoReconnect.
  • PRIMARY_PREFERRED: Read from the primary if available, otherwise readfrom a secondary.
  • SECONDARY: Read from a secondary. If no matching secondary is available,raise AutoReconnect.
  • SECONDARY_PREFERRED: Read from a secondary if available, otherwisefrom the primary.
  • NEAREST: Read from any available member.

Tag sets:

Replica-set members can be tagged according to anycriteria you choose. By default, PyMongo ignores tags whenchoosing a member to read from, but your read preference can be configured witha tag_sets parameter. tag_sets must be a list of dictionaries, eachdict providing tag values that the replica set member must match.PyMongo tries each set of tags in turn until it finds a set oftags with at least one matching member. For example, to prefer reads from theNew York data center, but fall back to the San Francisco data center, tag yourreplica set members according to their location and create aMongoClient like so:

  1. >>> from pymongo.read_preferences import Secondary
  2. >>> db = client.get_database(
  3. ...     'test', read_preference=Secondary([{'dc': 'ny'}, {'dc': 'sf'}]))
  4. >>> db.read_preference
  5. Secondary(tag_sets=[{'dc': 'ny'}, {'dc': 'sf'}])

MongoClient tries to find secondaries in New York, then San Francisco,and raises AutoReconnect if none are available. As anadditional fallback, specify a final, empty tag set, {}, which means “readfrom any member that matches the mode, ignoring tags.”

See read_preferences for more information.

Local threshold:

If multiple members match the read preference and tag sets, PyMongo readsfrom among the nearest members, chosen according to ping time. By default,only members whose ping times are within 15 milliseconds of the nearestare used for queries. You can choose to distribute reads among members withhigher latencies by setting localThresholdMS to a largernumber:

  1. >>> client = pymongo.MongoClient(
  2. ...     replicaSet='repl0',
  3. ...     readPreference='secondaryPreferred',
  4. ...     localThresholdMS=35)

In this case, PyMongo distributes reads among matching members within 35milliseconds of the closest member’s ping time.

Note

localThresholdMS is ignored when talking to areplica set through a mongos. The equivalent is the localThreshold commandline option.

Health Monitoring

When MongoClient is initialized it launches background threads tomonitor the replica set for changes in:

  • Health: detect when a member goes down or comes up, or if a different memberbecomes primary
  • Configuration: detect when members are added or removed, and detect changesin members’ tags
  • Latency: track a moving average of each member’s ping time

Replica-set monitoring ensures queries are continually routed to the propermembers as the state of the replica set changes.

mongos Load Balancing

An instance of MongoClient can be configuredwith a list of addresses of mongos servers:

  1. >>> client = MongoClient('mongodb://host1,host2,host3')

Each member of the list must be a single mongos server. Multihomed and roundrobin DNS addresses are not supported. The client continuouslymonitors all the mongoses’ availability, and its network latency to each.

PyMongo distributes operations evenly among the set of mongoses within itslocalThresholdMS (similar to how it distributes reads to secondariesin a replica set). By default the threshold is 15 ms.

The lowest-latency server, and all servers with latencies no more thanlocalThresholdMS beyond the lowest-latency server’s, receiveoperations equally. For example, if we have three mongoses:

  • host1: 20 ms
  • host2: 35 ms
  • host3: 40 ms

By default the localThresholdMS is 15 ms, so PyMongo uses host1 and host2evenly. It uses host1 because its network latency to the driver is shortest. Ituses host2 because its latency is within 15 ms of the lowest-latency server’s.But it excuses host3: host3 is 20ms beyond the lowest-latency server.

If we set localThresholdMS to 30 ms all servers are within the threshold:

  1. >>> client = MongoClient('mongodb://host1,host2,host3/?localThresholdMS=30')

Warning

Do not connect PyMongo to a pool of mongos instances through aload balancer. A single socket connection must always be routed to the samemongos instance for proper cursor support.