Three or more (3DC) deployments

Recommended Reading9 Techniques to Build Cloud-Native, Geo-Distributed SQL Apps with Low Latency highlights the various multi-DC deployment strategies (including 3DC deployments) for a distributed SQL database like YugabyteDB.

Three data center deployments of YugabyteDB are essentially a natural extension of the three availability zone (AZ) deployments documented in the Manual deployment section. Equal number of nodes are now placed in each data center of the three data centers. Inside a single data center, a multi-AZ deployment is recommended to ensure resilience against zone failures. This approach works fine for any odd number of AZs or data centers. Given YugabyteDB’s distributed consensus-based replication which requires majority quorum for continuous availability of write requests, deploying a single cluster across an even number of AZs or data centers is not recommended.

Example scenario

  • Create a three-node cluster with replication factor of 3.
    • Cloud will be aws and the three regions/AZs will be us-west/us-west-2a, us-east-1/us-east-1a, ap-northeast-1/ap-northeast-1a. One node will be placed in each region/AZ in such a way that one replica for each tablet also gets placed in each region/AZ.
    • Private IP addresses of the 3 nodes are 172.151.17.130, 172.151.17.220, and 172.151.17.140.
  • We have multiple data drives mounted on /home/centos/disk1, /home/centos/disk2.

Prerequisites

Follow the Checklist to ensure you have prepared the nodes for installing YugabyteDB.

Execute the following steps on each of the instances.

1. Install software

Follow the installation instructions to install YugabyteDB on each of the nodes.

2. Start YB-Masters

Run the yb-master server on each of the nodes as shown below. Note how multiple directories can be provided to the —fs_data_dirs option. Replace the —rpc_bind_addresses value with the private IP address of the host as well as the set the —placement_cloud,—placement_region and —placement_zone values appropriately.

  1. $ ./bin/yb-master \
  2.   --master_addresses 172.151.17.130:7100,172.151.17.220:7100,172.151.17.140:7100 \
  3.   --rpc_bind_addresses 172.151.17.130 \
  4.   --fs_data_dirs "/home/centos/disk1,/home/centos/disk2" \
  5.   --placement_cloud aws \
  6.   --placement_region us-west \
  7.   --placement_zone us-west-2a \
  8.   --leader_failure_max_missed_heartbeat_periods 10 \
  9.   >& /home/centos/disk1/yb-master.out &

Note that we also set the —leader_failure_max_missed_heartbeat_periods option to 10. This option specifies the maximum heartbeat periods that the leader can fail to heartbeat before the leader is considered to be failed. Since the data is geo-replicated across data centers, RPC latencies are expected to be higher. We use this flag to increase the failure detection interval in such a higher RPC latency deployment. Note that the total failure timeout is now 5 seconds since it is computed by multiplying —raft_heartbeat_interval_ms (default of 500ms) with —leader_failure_max_missed_heartbeat_periods (current value of 10).

For the full list of configuration options, see the YB-Master reference.

3. Start YB-TServers

Run the yb-tserver server on each node as shown below . Note that all of the master addresses have to be provided using the —tserver_master_addrs option. Replace the —rpc_bind_addresses value with the private IP address of the host as well as the set the placement_cloud,placement_region and placement_zone values appropriately.

  1. $ ./bin/yb-tserver \
  2.   --tserver_master_addrs 172.151.17.130:7100,172.151.17.220:7100,172.151.17.140:7100 \
  3.   --rpc_bind_addresses 172.151.17.130 \
  4.   --start_pgsql_proxy \
  5.   --pgsql_proxy_bind_address 172.151.17.130:5433 \
  6.   --cql_proxy_bind_address 172.151.17.130:9042 \
  7.   --fs_data_dirs "/home/centos/disk1,/home/centos/disk2" \
  8.   --placement_cloud aws \
  9.   --placement_region us-west \
  10.   --placement_zone us-west-2a \
  11.   --leader_failure_max_missed_heartbeat_periods 10 \
  12.   >& /home/centos/disk1/yb-tserver.out &

Note that we also set the —leader_failure_max_missed_heartbeat_periods option to 10. This option specifies the maximum heartbeat periods that the leader can fail to heartbeat before the leader is considered to be failed. Since the data is geo-replicated across data centers, RPC latencies are expected to be higher. We use this flag to increase the failure detection interval in such a higher RPC latency deployment. Note that the total failure timeout is now 5 seconds since it is computed by multiplying —raft_heartbeat_interval_ms (default of 500ms) with leader_failure_max_missed_heartbeat_periods (current value of 10).

For the full list of configuration flags, see the YB-TServer reference.

4. Set replica placement policy

The default replica placement policy when the cluster is first created is to treat all nodes as equal irrespective of the —placement_* configuration options. However, for the current deployment, we want to explicitly place one replica of each tablet in each region/AZ. The following command sets replication factor of 3 across us-west-2/us-west-2a, us-east-1/us-east-1a, ap-northeast-1/ap-northeast-1a leading to such a placement.

On any host running the yb-master, run the following command.

  1. $ ./bin/yb-admin \
  2.     --master_addresses 172.151.17.130:7100,172.151.17.220:7100,172.151.17.140:7100 \
  3.     modify_placement_info  \
  4.     aws.us-west.us-west-2a,aws.us-east-1.us-east-1a,aws.ap-northeast-1.ap-northeast-1a 3

Verify by running the following.

  1. $ curl -s http://<any-master-ip>:7000/cluster-config

And confirm that the output looks similar to what is shown below with —min_num_replicas set to 1 for each AZ.

  1. replication_info {
  2.   live_replicas {
  3.     num_replicas: 3
  4.     placement_blocks {
  5.       cloud_info {
  6.         placement_cloud: "aws"
  7.         placement_region: "us-west"
  8.         placement_zone: "us-west-2a"
  9.       }
  10.       min_num_replicas: 1
  11.     }
  12.     placement_blocks {
  13.       cloud_info {
  14.         placement_cloud: "aws"
  15.         placement_region: "us-east-1"
  16.         placement_zone: "us-east-1a"
  17.       }
  18.       min_num_replicas: 1
  19.     }
  20.     placement_blocks {
  21.       cloud_info {
  22.         placement_cloud: "aws"
  23.         placement_region: "ap-northeast-1"
  24.         placement_zone: "ap-northeast-1a"
  25.       }
  26.       min_num_replicas: 1
  27.     }
  28.   }
  29. }

5. Verify deployment

Use the ysqlsh (for YSQL API) or cqlsh (for YCQL API) shells to test connectivity to the cluster.