Upgrade MongoDB to 2.4

In the general case, the upgrade from MongoDB 2.2 to 2.4 is abinary-compatible “drop-in” upgrade: shut down the mongodinstances and replace them with mongod instances running2.4. However, before you attempt any upgrade please familiarizeyourself with the content of this document, particularly the procedurefor upgrading sharded clusters and theconsiderations for reverting to 2.2 after running 2.4.

Upgrade Recommendations and Checklist

When upgrading, consider the following:

  • For all deployments using authentication, upgrade thedrivers (i.e. client libraries), before upgrading themongod instance or instances.

  • To upgrade to 2.4 sharded clusters must upgrade following themeta-data upgrade procedure.

  • If you’re using 2.2.0 and running with authorization enabled, youwill need to upgrade first to 2.2.1 and then upgrade to 2.4. SeeRolling Upgrade Limitation for 2.2.0 Deployments Running with auth Enabled.

  • If you have system.users documents(i.e. for authorization) that you created before 2.4 you must_ensure that there are no duplicate values for the user field inthe system.users collection in_any database. If you do have documents with duplicate userfields, you must remove them before upgrading.

See Security Enhancements for more information.

Upgrade Standalone mongod Instance to MongoDB 2.4

Upgrade a Replica Set from MongoDB 2.2 to MongoDB 2.4

You can upgrade to 2.4 by performing a “rolling”upgrade of the set by upgrading the members individually while theother members are available to minimize downtime. Use the followingprocedure:

  • Upgrade the secondary members of the set one at a time byshutting down the mongod and replacing the 2.2 binarywith the 2.4 binary. After upgrading a mongod instance,wait for the member to recover to SECONDARY statebefore upgrading the next instance.To check the member’s state, issue rs.status() in themongo shell.

  • Use the mongo shell method rs.stepDown() tostep down the primary to allow the normal failover procedure. rs.stepDown()expedites the failover procedure and is preferable to shutting downthe primary directly.

Once the primary has stepped down and another member has assumedPRIMARY state, as observed in the output ofrs.status(), shut down the previous primary and replacemongod binary with the 2.4 binary and start the newprocess.

Note

Replica set failover is not instant but willrender the set unavailable to read or accept writesuntil the failover process completes. Typically this takes10 seconds or more. You may wish to plan the upgrade duringa predefined maintenance window.

Upgrade a Sharded Cluster from MongoDB 2.2 to MongoDB 2.4

Important

Only upgrade sharded clusters to 2.4 if all membersof the cluster are currently running instances of 2.2. The onlysupported upgrade path for sharded clusters running 2.0 is via 2.2.

Overview

Upgrading a sharded cluster from MongoDB version 2.2 to 2.4(or 2.3) requires that you run a 2.4 mongoswith the —upgrade option, described inthis procedure. The upgrade process does not require downtime.

The upgrade to MongoDB 2.4 adds epochs to the meta-data for allcollections and chunks in the existing cluster. MongoDB 2.2 processesare capable of handling epochs, even though 2.2 did not requirethem. This procedure applies only to upgrades from version 2.2. Earlierversions of MongoDB do not correctly handle epochs. SeeCluster Meta-data Upgrade for more information.

After completing the meta-data upgrade you can fully upgrade thecomponents of the cluster. With the balancer disabled:

  • Upgrade all mongos instances in the cluster.
  • Upgrade all 3 mongod config server instances.
  • Upgrade the mongod instances for each shard, one at atime.

See Upgrade Sharded Cluster Components for more information.

Cluster Meta-data Upgrade

Considerations

Beware of the following properties of the cluster upgrade process:

  • Before you start the upgrade, ensure that the amount of free space onthe filesystem for the config database is at least 4 to 5 times the amount ofspace currently used by the config database data files.

Additionally, ensure that all indexes in the config database are {v:1} indexes. If a criticalindex is a {v:0} index, chunk splits can fail due to known issueswith the {v:0} format. {v:0} indexes are present on clusters created withMongoDB 2.0 or earlier.

The duration of the metadata upgrade depends on the network latencybetween the node performing the upgrade and the three config servers.Ensure low latency between the upgrade process and the config servers.

The upgraded config database will require more storage space thanbefore, to make backup and working copies of theconfig.chunks and config.collections collections.As always, if storage requirements increase, the mongodmight need to pre-allocate additional data files. SeeHow can I get information on the storage use of a database? for more information.

Meta-data Upgrade Procedure

Changes to the meta-data format for sharded clusters, stored in theconfig database, require a specialmeta-data upgrade procedure when moving to 2.4.

Do not perform operations that modify meta-data while performing thisprocedure. See Upgrade a Sharded Cluster from MongoDB 2.2 to MongoDB 2.4 for examples of prohibitedoperations.

  • Before you start the upgrade, ensure that the amount of free space onthe filesystem for the config database is at least 4 to 5 times the amount ofspace currently used by the config database data files.

Additionally, ensure that all indexes in the config database are {v:1} indexes. If a criticalindex is a {v:0} index, chunk splits can fail due to known issueswith the {v:0} format. {v:0} indexes are present on clusters created withMongoDB 2.0 or earlier.

The duration of the metadata upgrade depends on the network latencybetween the node performing the upgrade and the three config servers.Ensure low latency between the upgrade process and the config servers.

To check the version of your indexes, use db.collection.getIndexes().

If any index on the config database is {v:0}, you shouldrebuild those indexes by connecting to the mongos andeither: rebuild all indexes using thedb.collection.reIndex() method, or drop and rebuildspecific indexes using db.collection.dropIndex() and thendb.collection.ensureIndex(). If you need to upgrade the_id index to {v:1} use db.collection.reIndex().

You may have {v:0} indexes on other databases in the cluster.

Optional

For additional security during the upgrade, you can make abackup of the config database using mongodump orother backup tools.

  • Ensure there are no version 2.0 mongod ormongos processes still active in the shardedcluster. The automated upgrade process checks for 2.0 processes,but network availability can prevent a definitive check. Wait 5minutes after stopping or upgrading version 2.0 mongosprocesses to confirm that none are still active.

  • Start a single 2.4 mongos process withconfigDB pointing to the sharded cluster’s configservers and with the —upgradeoption. The upgrade process happens before theprocess becomes a daemon (i.e. before—fork.)

You can upgrade an existingmongos instance to 2.4 or you can start a new _mongos_instance that can reach all config servers if you need to avoidreconfiguring a production mongos.

Start the mongos with a command that resembles thefollowing:

  1. mongos --configdb <config servers> --upgrade

Without the —upgrade option 2.4mongos processes will fail to start until the upgradeprocess is complete.

The upgrade will prevent any chunk moves or splits from occurringduring the upgrade process. If there are very many shardedcollections or there are stale locks held by other failed processes,acquiring the locks for all collections can takeseconds or minutes. See the log for progress updates.

  • When the mongos process starts successfully, the upgrade iscomplete. If the mongos process fails to start, check thelog for more information.

If the mongos terminates or loses its connection to theconfig servers during the upgrade, you may always safely retry theupgrade.

However, if the upgrade failed during the short critical section,the mongos will exit and report that the upgrade willrequire manual intervention. To continue the upgrade process, youmust follow the Resync after an Interruption of the Critical Section procedure.

Optional

If the mongos logs show the upgrade waiting for theupgrade lock, a previous upgrade process may still be active ormay have ended abnormally. After 15 minutes of no remoteactivity mongos will force the upgrade lock. If youcan verify that there are no running upgrade processes, you mayconnect to a 2.2 mongos process and force the lockmanually:

  1. mongo <mongos.example.net>
  1. db.getMongo().getCollection("config.locks").findOne({ _id : "configUpgrade" })

If the process specified in the process field of this documentis verifiably offline, run the following operation to force thelock.

  1. db.getMongo().getCollection("config.locks").update({ _id : "configUpgrade" }, { $set : { state : 0 } })

It is always more safe to wait for the mongos toverify that the lock is inactive, if you have any doubts aboutthe activity of another upgrade operation. In addition to theconfigUpgrade, the mongos may need to wait forspecific collection locks. Do not force the specific collectionlocks.

  • Upgrade and restart other mongos processes in thesharded cluster, without the —upgradeoption.

See Upgrade Sharded Cluster Components for more information.

Once you have upgraded, do not introduce version 2.0 MongoDBprocesses into the sharded cluster. This can reintroduce old meta-dataformats into the config servers. The meta-data change made by thisupgrade process will help prevent errors caused by cross-versionincompatibilities in future versions of MongoDB.

Resync after an Interruption of the Critical Section

During the short critical section of the upgrade that applies changesto the meta-data, it is unlikely but possible that a networkinterruption can prevent all three config servers from verifying ormodifying data. If this occurs, the config servers must be re-synced, and there may be problemsstarting new mongos processes. The sharded clusterwill remain accessible, but avoid all cluster meta-data changes untilyou resync the config servers. Operations that change meta-data include:adding shards, dropping databases, and dropping collections.

Note

Only perform the following procedure if something (e.g. network,power, etc.) interrupts the upgrade process during the shortcritical section of the upgrade. Remember, you may always safelyattempt the meta data upgrade procedure.

To resync the config servers:

  • Turn off the balancer in thesharded cluster and stop all meta-data operations. If you are in themiddle of an upgrade process (Upgrade a Sharded Cluster from MongoDB 2.2 to MongoDB 2.4), youhave already disabled the balancer.

  • Shut down two of the three config servers, preferably the last two listedin the configDB string. For example, if your configDBstring is configA:27019,configB:27019,configC:27019, shut downconfigB and configC. Shutting down the last two config serversensures that most mongos instances will haveuninterrupted access to cluster meta-data.

  • mongodump the data files of the active config server(configA).

  • Move the data files of the deactivated config servers (configBand configC) to a backup location.

  • Create new, empty data directories.

  • Restart the disabled config servers with —dbpathpointing to the now-empty data directory and —portpointing to an alternate port (e.g. 27020).

  • Use mongorestore to repopulate the data files on thedisabled documents from the activeconfig server (configA) to the restarted config servers on the newport (configB:27020,configC:27020). These config servers are nowre-synced.

  • Restart the restored config servers on the old port, resetting theport back to the old settings (configB:27019 and configC:27019).

  • In some cases connection pooling may cause spurious failures, asthe mongos disables old connections only after attempteduse. 2.4 fixes this problem, but to avoid this issue in version2.2, you can restart all mongos instances (one-by-one,to avoid downtime) and use the rs.stepDown() methodbefore restarting each of the shard replica setprimaries.

  • The sharded cluster is now fully resynced; however before youattempt the upgrade process again, you must manually reset theupgrade state using a version 2.2 mongos. Begin byconnecting to the 2.2 mongos with the mongoshell:

  1. mongo <mongos.example.net>

Then, use the following operation to reset the upgrade process:

  1. db.getMongo().getCollection("config.version").update({ _id : 1 }, { $unset : { upgradeState : 1 } })

Upgrade Sharded Cluster Components

After you have successfully completed the meta-data upgrade processdescribed in Meta-data Upgrade Procedure, and the 2.4mongos instance starts, you can upgrade the other processesin your MongoDB deployment.

While the balancer is still disabled, upgrade the components of yoursharded cluster in the following order:

  • Upgrade all mongos instances in the cluster, in anyorder.
  • Upgrade all 3 mongod config server instances, upgradingthe first system in the mongos —configdb argumentlast.
  • Upgrade each shard, one at a time, upgrading the mongodsecondaries before running replSetStepDown andupgrading the primary of each shard.

When this process is complete, you can now re-enable thebalancer.

Rolling Upgrade Limitation for 2.2.0 Deployments Running with auth Enabled

MongoDB cannot support deployments that mix 2.2.0 and 2.4.0, orgreater, components. MongoDB version 2.2.1 and later processes _can_exist in mixed deployments with 2.4-series processes. Therefore youcannot perform a rolling upgrade from MongoDB 2.2.0 to MongoDB2.4.0. To upgrade a cluster with 2.2.0 components, use one of thefollowing procedures.

  • Perform a rolling upgrade of all 2.2.0 processes to the latest2.2-series release (e.g. 2.2.3) so that there are no processes inthe deployment that predate 2.2.1. When there are no 2.2.0processes in the deployment, perform a rolling upgrade to 2.4.0.
  • Stop all processes in the cluster. Upgrade all processes to a2.4-series release of MongoDB, and start all processes at the sametime.

Upgrade from 2.3 to 2.4

If you used a mongod from the 2.3 or 2.4-rc (releasecandidate) series, you can safely transition these databases to 2.4.0or later; however, if you created 2dsphere or text indexesusing a mongod before v2.4-rc2, you will need to rebuildthese indexes. For example:

  1. db.records.dropIndex( { loc: "2dsphere" } )
  2. db.records.dropIndex( "records_text" )
  3.  
  4. db.records.ensureIndex( { loc: "2dsphere" } )
  5. db.records.ensureIndex( { records: "text" } )

Downgrade MongoDB from 2.4 to Previous Versions

For some cases the on-disk format of data files used by 2.4 and 2.2mongod is compatible, and you can upgrade and downgrade ifneeded. However, several new features in 2.4 are incompatible withprevious versions:

  • 2dsphere indexes are incompatible with 2.2 and earliermongod instances.
  • text indexes are incompatible with 2.2 and earliermongod instances.
  • using a hashed index as a shard key are incompatible with 2.2 andearlier mongos instances.
  • hashed indexes are incompatible with 2.0 and earliermongod instances.

Important

Collections sharded using hashed shard keys, shouldnot use 2.2 mongod instances, which cannot correctlysupport cluster operations for these collections.

If you completed the meta-data upgrade for a sharded cluster, you can safely downgrade to 2.2 MongoDBprocesses. Do not use 2.0 processes after completing the upgradeprocedure.

Note

In sharded clusters, once you have completed the meta-data upgradeprocedure, you cannot use 2.0mongod or mongos instances in the samecluster.

If you complete the meta-data upgrade, you can safely downgradecomponents in any order. When upgrade again, alwaysupgrade mongos instances before mongod instances.

Do not create 2dsphere or text indexes in a clusterthat has 2.2 components.

Considerations and Compatibility

If you upgrade to MongoDB 2.4, and then need to run MongoDB 2.2 withthe same data files, consider the following limitations.

  • If you use a hashed index as the shard key index, which is onlypossible under 2.4 you will not be able to query data in thissharded collection. Furthermore, a 2.2 mongos cannotproperly route an insert operation for a collections sharded using ahashed index for the shard key index: any data that you insertusing a 2.2 mongos, will not arrive on the correct shardand will not be reachable by future queries.
  • If you never create an 2dsphere or text index, you canmove between a 2.4 and 2.2 mongod for a given data set;however, after you create the first 2dsphere or text indexwith a 2.4 mongod you will need to run a 2.2mongod with the —upgradeoption and drop any 2dsphere or text index.

Upgrade and Downgrade Procedures

Basic Downgrade and Upgrade

Except as described below, moving between 2.2 and 2.4 is a drop-inreplacement:

  1. mongod --dbpath /var/mongod/data --shutdown

Replace /var/mongod/data with your MongoDB dbPath.

  • start the new mongod processes with the samedbPath setting, for example:
  1. mongod --dbpath /var/mongod/data

Replace /var/mongod/data with your MongoDB dbPath.

Downgrade to 2.2 After Creating a 2dsphere or text Index

If you have created 2dsphere or text indexes while running a2.4 mongod instance, you can downgrade at any time, bystarting the 2.2 mongod with the —upgrade option as follows:

  1. mongod --dbpath /var/mongod/data/ --upgrade

Then, you will need to drop any existing 2dsphere or textindexes using db.collection.dropIndex(), for example:

  1. db.records.dropIndex( { loc: "2dsphere" } )
  2. db.records.dropIndex( "records_text" )

Warning

—upgrade will runrepairDatabase on any database where you have createda 2dsphere or text index, which will rebuild _all_indexes.

Troubleshooting Upgrade/Downgrade Operations

If you do not use —upgrade, when youattempt to start a 2.2 mongod and you have created a2dsphere or text index, mongod will return thefollowing message:

  1. 'need to upgrade database index_plugin_upgrade with pdfile version 4.6, new version: 4.5 Not upgrading, exiting'

While running 2.4, to check the data file version of a MongoDBdatabase, use the following operation in the shell:

  1. db.getSiblingDB('<databasename>').stats().dataFileVersion

The major data file [1] version for both 2.2 and 2.4 is4, the minor data file version for 2.2 is 5 and the minor datafile version for 2.4 is 6 after you create a 2dsphere ortext index.

[1]The data file version (i.e. pdfile version)is independent and unrelated to the release version of MongoDB.