Deploy a Geographically Redundant Replica Set

Overview

This tutorial outlines the process for deploying a replica setwith members in multiple locations. Thetutorial addresses three-member replica sets and five-member replicasets. If you have an even number of replica set members, add an arbiterto deploy an odd number replica set.

For more information on distributed replica sets, seeReplica Sets Distributed Across Two or More Data Centers. Seealso Replica Set Deployment Architectures and see Replication.

Considerations

Architecture

In production, deploy each member of the replica set to its own machineand if possible bind to the standard MongoDB port of 27017.

See Replica Set Deployment Architectures for more information.

Hostnames

Tip

When possible, use a logical DNS hostname instead of an ip address,particularly when configuring replica set members or sharded clustermembers. The use of logical DNS hostnames avoids configurationchanges due to ip address changes.

IP Binding

Use the bind_ip option to ensure that MongoDB listens forconnections from applications on configured addresses.

Starting in MongoDB 3.6, MongoDB binaries, mongod andmongos, bind to localhost by default. If thenet.ipv6 configuration file setting or the —ipv6command line option is set for the binary, the binary additionally bindsto the localhost IPv6 address.

Previously, starting from MongoDB 2.6, only the binaries from theofficial MongoDB RPM (Red Hat, CentOS, Fedora Linux, and derivatives)and DEB (Debian, Ubuntu, and derivatives) packages bind to localhost bydefault.

When bound only to the localhost, these MongoDB 3.6 binaries can onlyaccept connections from clients (including the mongo shell,other members in your deployment for replica sets and sharded clusters)that are running on the same machine. Remote clients cannot connect tothe binaries bound only to localhost.

To override and bind to other ip addresses, you can use thenet.bindIp configuration file setting or the—bind_ip command-line option to specify a list of hostnames or ipaddresses.

Warning

Before binding to a non-localhost (e.g. publicly accessible)IP address, ensure you have secured your cluster from unauthorizedaccess. For a complete list of security recommendations, seeSecurity Checklist. At minimum, considerenabling authentication andhardening network infrastructure.

For example, the following mongod instance binds to boththe localhost and the hostname My-Example-Associated-Hostname, which isassociated with the ip address 198.51.100.1:

  1. mongod --bind_ip localhost,My-Example-Associated-Hostname

In order to connect to this instance, remote clients must specifythe hostname or its associated ip address 198.51.100.1:

  1. mongo --host My-Example-Associated-Hostname
  2.  
  3. mongo --host 198.51.100.1

Connectivity

Ensure that network traffic can pass securely between all members of the set andall clients in the network .

Consider the following:

  • Establish a virtual private network. Ensure that your network topologyroutes all traffic between members within a single site over the localarea network.
  • Configure access control to prevent connections from unknown clientsto the replica set.
  • Configure networking and firewall rules so that incoming and outgoingpackets are permitted only on the default MongoDB port and only fromwithin your deployment. See the IP Binding considerations.

Ensure that each member of a replica set is accessible byway of resolvable DNS or hostnames. You should either configure yourDNS names appropriately or set up your systems’ /etc/hosts file toreflect this configuration.

Each member must be able to connect to every other member. Forinstructions on how to check your connection, seeTest Connections Between all Members.

Configuration

Create the directory where MongoDB stores data files before deployingMongoDB.

Specify the mongod configuration in a configurationfile stored in /etc/mongod.confor a related location.

For more information about configuration options, seeConfiguration File Options.

Distribution of the Members

If possible, use an odd number of data centers, and choose adistribution of members that maximizes the likelihood that even with aloss of a data center, the remaining replica set members can form amajority or at minimum, provide a copy of your data.

Voting Members

Never deploy more than seven voting members.

Prerequisites

For all configurations in this tutorial, deploy each replica set memberon a separate system. Although you may deploy more than one replica set member on asingle system, doing so reduces the redundancy and capacityof the replica set. Such deployments are typically for testingpurposes.

This tutorial assumes you have installed MongoDB on each system thatwill be part of your replica set. If you have not already installedMongoDB, see the installation tutorials.

Procedures

Deploy a Geographically Redundant Three-Member Replica Set

Tip

When possible, use a logical DNS hostname instead of an ip address,particularly when configuring replica set members or sharded clustermembers. The use of logical DNS hostnames avoids configurationchanges due to ip address changes.

For a geographically redundant three-member replica set deployment, youmust decide how to distribute your system. Some possible distributionsfor the three members are:

  • Across Three Data Centers: One member to each site.
  • Across Two Data Centers: Two members to Site A and one member to SiteB. If one of the members of the replica set is an arbiter, distributethe arbiter to Site A with a data-bearing member.

Note

Distributing replica set members across two data centers providesbenefit over a single data center. In a two data centerdistribution,

  • If one of the data centers goes down, the data is still availablefor reads unlike a single data center distribution.
  • If the data center with a minority of the members goes down, thereplica set can still serve write operations as well as readoperations.
  • However, if the data center with the majority of the members goesdown, the replica set becomes read-only.

If possible, distribute members across at least three data centers.For config server replica sets (CSRS), the best practice is todistribute across three (or more depending on the number of members)centers. If the cost of the third data center is prohibitive, onedistribution possibility is to evenly distribute the data bearingmembers across the two data centers and store the remaining member(either a data bearing member or an arbiter to ensure odd numberof members) in the cloud if your company policy allows.

Start each member of the replica set with the appropriate options.

For each member, start a mongod instance with thefollowing settings:

If your application connects to more than one replica set, each setshould have a distinct name. Some drivers group replica setconnections by replica set name.

  • Set net.bindIp option to the hostname/ip or a comma-delimitedlist of hostnames/ips, and

  • Set any other settings as appropriate for your deployment.

In this tutorial, the three mongod instances areassociated with the following hosts:

Replica Set MemberHostname
Member 0mongodb0.example.net
Member 1mongodb1.example.net
Member 2mongodb2.example.net

The following example specifies the replica set name and the ipbinding through the —replSet and —bind_ipcommand-line options:

Warning

Before binding to a non-localhost (e.g. publicly accessible)IP address, ensure you have secured your cluster from unauthorizedaccess. For a complete list of security recommendations, seeSecurity Checklist. At minimum, considerenabling authentication andhardening network infrastructure.

  1. mongod --replSet "rs0" --bind_ip localhost,<hostname(s)|ip address(es)>

For <hostname(s)|ip address(es)>, specify the hostname(s) and/orip address(es) for your mongod instance that remoteclients (including the other members of the replica set) can use toconnect to the instance.

Alternatively, you can also specify the replica set name and the ip addresses in a configuration file:

  1. replication:
  2.    replSetName: "rs0"
  3. net:
  4.    bindIp: localhost,<hostname(s)|ip address(es)>

To start mongod with a configuration file, specify theconfiguration file’s path with the —config option:

  1. mongod --config <path-to-config>

In production deployments, you can configure a init scriptto manage this process. Init scripts are beyond the scope of thisdocument.

Connect a mongo shell to one of the mongod instances.

From the same machine where one of the mongod is running(in this tutorial, mongodb0.example.net), start the mongoshell. To connect to the mongod listening to localhost onthe default port of 27017, simply issue:

  1. mongo

Depending on your path, you may need to specify the path to themongo binary.

Initiate the replica set.

From the mongo shell, run rs.initiate() onreplica set member 0.

Important

Run rs.initiate() on just one and only onemongod instance for the replica set.

Tip

When possible, use a logical DNS hostname instead of an ip address,particularly when configuring replica set members or sharded clustermembers. The use of logical DNS hostnames avoids configurationchanges due to ip address changes.

  1. rs.initiate( {
  2.    _id : "rs0",
  3.    members: [
  4.       { _id: 0, host: "mongodb0.example.net:27017" },
  5.       { _id: 1, host: "mongodb1.example.net:27017" },
  6.       { _id: 2, host: "mongodb2.example.net:27017" }
  7.    ]
  8. })

MongoDB initiates a replica set, usingthe default replica set configuration.

View the replica set configuration.

Use rs.conf() to display the replica set configurationobject:

  1. rs.conf()

The replica set configuration object resembles the following:

  1. {
  2.    "_id" : "rs0",
  3.    "version" : 1,
  4.    "protocolVersion" : NumberLong(1),
  5.    "members" : [
  6.       {
  7.          "_id" : 0,
  8.          "host" : "mongodb0.example.net:27017",
  9.          "arbiterOnly" : false,
  10.          "buildIndexes" : true,
  11.          "hidden" : false,
  12.          "priority" : 1,
  13.          "tags" : {
  14.  
  15.          },
  16.          "slaveDelay" : NumberLong(0),
  17.          "votes" : 1
  18.       },
  19.       {
  20.          "_id" : 1,
  21.          "host" : "mongodb1.example.net:27017",
  22.          "arbiterOnly" : false,
  23.          "buildIndexes" : true,
  24.          "hidden" : false,
  25.          "priority" : 1,
  26.          "tags" : {
  27.  
  28.          },
  29.          "slaveDelay" : NumberLong(0),
  30.          "votes" : 1
  31.       },
  32.       {
  33.          "_id" : 2,
  34.          "host" : "mongodb2.example.net:27017",
  35.          "arbiterOnly" : false,
  36.          "buildIndexes" : true,
  37.          "hidden" : false,
  38.          "priority" : 1,
  39.          "tags" : {
  40.  
  41.          },
  42.          "slaveDelay" : NumberLong(0),
  43.          "votes" : 1
  44.       }
  45.  
  46.    ],
  47.    "settings" : {
  48.       "chainingAllowed" : true,
  49.       "heartbeatIntervalMillis" : 2000,
  50.       "heartbeatTimeoutSecs" : 10,
  51.       "electionTimeoutMillis" : 10000,
  52.       "catchUpTimeoutMillis" : -1,
  53.       "getLastErrorModes" : {
  54.  
  55.       },
  56.       "getLastErrorDefaults" : {
  57.          "w" : 1,
  58.          "wtimeout" : 0
  59.       },
  60.       "replicaSetId" : ObjectId("585ab9df685f726db2c6a840")
  61.    }
  62. }

Optional. Configure the member eligibility for becoming primary.

In some cases, you may prefer that the members in one data center beelected primary before the members in the other data centers. You canmodify the priority of the members such that themembers in the one data center has higherpriority than the members in the other datacenters.

Some members of the replica set, such as members that have networkingrestraint or limited resources, should not be able to become primaryin a failover. Configure members that should not becomeprimary to have priority 0.

For example, to lower the relative eligibility of the memberlocated in one of the sites (in this example,mongodb2.example.net), set the member’s priority to 0.5.

  • View the replica set configuration to determine themembers array position for the member. Keepin mind the array position is not the same as the _id:
  1. rs.conf()
  • Copy the replica set configuration object to a variable (to cfg inthe example below). Then, in the variable, set the correct priority forthe member. Then pass the variable to rs.reconfig() to updatethe replica set configuration.

For example, to set priority for the third member in the array (i.e.,the member at position 2), issue the following sequence of commands:

  1. cfg = rs.conf()
  2. cfg.members[2].priority = 0.5
  3. rs.reconfig(cfg)

Note

The rs.reconfig() shell method can force the currentprimary to step down, causing an election. When the primary stepsdown, all clients will disconnect. This is the intended behavior.While most elections complete within a minute, always make sureany replica configuration changes occur during scheduledmaintenance periods.

After these commands return, you have a geographically redundantthree-member replica set.

Ensure that the replica set has a primary.

Use rs.status() to identify the primary in the replica set.

Deploy a Geographically Redundant Five-Member Replica Set

Tip

When possible, use a logical DNS hostname instead of an ip address,particularly when configuring replica set members or sharded clustermembers. The use of logical DNS hostnames avoids configurationchanges due to ip address changes.

For a geographically redundant five-member replica set deployment, youmust decide how to distribute your system. Some possible distributionsfor the five members are:

  • Across Three Data Centers: Two members in Site A, two members in SiteB, one member in Site C.
  • Across Four Data Centers: Two members in one site, and one member inthe other three sites.
  • Across Five Data Centers: One member in each site.
  • Across Two Data Centers: Three members in Site A and two members inSite B. If possible, avoid distributing config server replica set across onlytwo data centers.

Note

Distributing replica set members across two data centers providesbenefit over a single data center. In a two data centerdistribution,

  • If one of the data centers goes down, the data is still availablefor reads unlike a single data center distribution.
  • If the data center with a minority of the members goes down, thereplica set can still serve write operations as well as readoperations.
  • However, if the data center with the majority of the members goesdown, the replica set becomes read-only.

If possible, distribute members across at least three data centers.For config server replica sets (CSRS), the best practice is todistribute across three (or more depending on the number of members)centers. If the cost of the third data center is prohibitive, onedistribution possibility is to evenly distribute the data bearingmembers across the two data centers and store the remaining member(either a data bearing member or an arbiter to ensure odd numberof members) in the cloud if your company policy allows.

The following five-member replica set includes an arbiter.

Start each member of the replica set with the appropriate options.

For each member, start a mongod instance with thefollowing settings:

If your application connects to more than one replica set, each setshould have a distinct name. Some drivers group replica setconnections by replica set name.

  • Set net.bindIp option to the hostname/ip address or acomma-delimited list of hostnames/ip addresses, and

  • Set any other settings as appropriate for your deployment.

In this tutorial, the five mongod instances areassociated with the following hosts:

Replica Set MemberHostname
Member 0mongodb0.example.net
Member 1mongodb1.example.net
Member 2mongodb2.example.net
Member 3mongodb3.example.net
Member 4mongodb4.example.net

The following example specifies the replica set name and the ipbinding through the —replSet and —bind_ipcommand-line options:

Warning

Before binding to a non-localhost (e.g. publicly accessible)IP address, ensure you have secured your cluster from unauthorizedaccess. For a complete list of security recommendations, seeSecurity Checklist. At minimum, considerenabling authentication andhardening network infrastructure.

  1. mongod --replSet "rs0" --bind_ip localhost,<hostname(s)|ip address(es)>

For <hostname(s)|ip address(es)>, specify the hostname(s) and/orip address(es) for your mongod instance that remoteclients (including the other members of the replica set) can use toconnect to the instance.

Alternatively, you can also specify the replica set name and the hostnames/ipaddresses in aconfiguration file:

  1. replication:
  2.    replSetName: "rs0"
  3. net:
  4.    bindIp: localhost,<hostname(s)|ip address(es)>

To start mongod with a configuration file, specify theconfiguration file’s path with the —config option:

  1. mongod --config <path-to-config>

In production deployments, you can configure a init scriptto manage this process. Init scripts are beyond the scope of thisdocument.

Connect a mongo shell to one of the mongod instances.

From the same machine where one of the mongod is running(in this tutorial, mongodb0.example.net), start the mongoshell. To connect to the mongod listening to localhost onthe default port of 27017, simply issue:

  1. mongo

Depending on your path, you may need to specify the path to themongo binary.

Initiate the replica set.

From the mongo shell, run rs.initiate() onreplica set member 0.

Important

Run rs.initiate() on just one and only onemongod instance for the replica set.

  1. rs.initiate( {
  2.    _id : "rs0",
  3.    members: [
  4.       { _id: 0, host: "mongodb0.example.net:27017" },
  5.       { _id: 1, host: "mongodb1.example.net:27017" },
  6.       { _id: 2, host: "mongodb2.example.net:27017" },
  7.       { _id: 3, host: "mongodb3.example.net:27017" },
  8.       { _id: 4, host: "mongodb4.example.net:27017", arbiterOnly: true }
  9.    ]
  10. })

The last member is added as an arbiter.

For the following MongoDB versions, pv1 increases the likelihoodof w:1 rollbacks compared to pv0(no longer supported in MongoDB 4.0+) for replica sets with arbiters:

  • MongoDB 3.4.1
  • MongoDB 3.4.0
  • MongoDB 3.2.11 or earlier

See Replica Set Protocol Version.

View the replica set configuration.

Use rs.conf() to display the replica set configurationobject:

  1. rs.conf()

The replica set configuration object resembles the following:

  1. {
  2.    "_id" : "rs0",
  3.    "version" : 1,
  4.    "protocolVersion" : NumberLong(1),
  5.    "members" : [
  6.       {
  7.          "_id" : 0,
  8.          "host" : "mongodb0.example.net:27017",
  9.          "arbiterOnly" : false,
  10.          "buildIndexes" : true,
  11.          "hidden" : false,
  12.          "priority" : 1,
  13.          "tags" : {
  14.  
  15.          },
  16.          "slaveDelay" : NumberLong(0),
  17.          "votes" : 1
  18.       },
  19.       {
  20.          "_id" : 1,
  21.          "host" : "mongodb1.example.net:27017",
  22.          "arbiterOnly" : false,
  23.          "buildIndexes" : true,
  24.          "hidden" : false,
  25.          "priority" : 1,
  26.          "tags" : {
  27.  
  28.          },
  29.          "slaveDelay" : NumberLong(0),
  30.          "votes" : 1
  31.       },
  32.       {
  33.          "_id" : 2,
  34.          "host" : "mongodb2.example.net:27017",
  35.          "arbiterOnly" : false,
  36.          "buildIndexes" : true,
  37.          "hidden" : false,
  38.          "priority" : 1,
  39.          "tags" : {
  40.  
  41.          },
  42.          "slaveDelay" : NumberLong(0),
  43.          "votes" : 1
  44.       }
  45.  
  46.    ],
  47.    "settings" : {
  48.       "chainingAllowed" : true,
  49.       "heartbeatIntervalMillis" : 2000,
  50.       "heartbeatTimeoutSecs" : 10,
  51.       "electionTimeoutMillis" : 10000,
  52.       "catchUpTimeoutMillis" : -1,
  53.       "getLastErrorModes" : {
  54.  
  55.       },
  56.       "getLastErrorDefaults" : {
  57.          "w" : 1,
  58.          "wtimeout" : 0
  59.       },
  60.       "replicaSetId" : ObjectId("585ab9df685f726db2c6a840")
  61.    }
  62. }

Optional. Configure the member eligibility for becoming primary.

In some cases, you may prefer that the members in one data center beelected primary before the members in the other data centers. You canmodify the priority of the members such that themembers in the one data center has higherpriority than the members in the other datacenters.

Some members of the replica set, such as members that have networkingrestraint or limited resources, should not be able to become primaryin a failover. Configure members that should not becomeprimary to have priority 0.

For example, to lower the relative eligibility of the memberlocated in one of the sites (in this example,mongodb2.example.net), set the member’s priority to 0.5.

  • View the replica set configuration to determine themembers array position for the member. Keepin mind the array position is not the same as the _id:
  1. rs.conf()
  • Copy the replica set configuration object to a variable (to cfg inthe example below). Then, in the variable, set the correct priority forthe member. Then pass the variable to rs.reconfig() to updatethe replica set configuration.

For example, to set priority for the third member in the array (i.e.,the member at position 2), issue the following sequence of commands:

  1. cfg = rs.conf()
  2. cfg.members[2].priority = 0.5
  3. rs.reconfig(cfg)

Note

The rs.reconfig() shell method can force the currentprimary to step down, causing an election. When the primary stepsdown, all clients will disconnect. This is the intended behavior.While most elections complete within a minute, always make sureany replica configuration changes occur during scheduledmaintenance periods.

After these commands return, you have a geographically redundantfive-member replica set.

Ensure that the replica set has a primary.

Use rs.status() to identify the primary in the replica set.