Global Distribution

AttentionThis page documents an earlier version. Go to the latest (v2.1)version.

YugabyteDB can easily be deployed in a globally distributed manner to serve application queries from the region closest to the end users with low latencies as well as to survive any outages to ensure high availability.

This tutorial will simulate AWS regions on a local machine. First, we will deploy YugabyteDB in the us-west-2 region across multiple availability zones (a, b, c). We will start a key-value workload against this universe. Next, we will change this setup to run across multiple geographic regions in US East (us-east-1) and Tokyo (ap-northeast-1), with the workload running uninterrupted during the entire transition.

If you haven’t installed YugabyteDB yet, do so first by following the Quick Start guide.

1. Create a multi-zone universe in US West

If you have a previously running local universe, destroy it using the following.

  1. $ ./bin/yb-ctl destroy

Start a new local universe with replication factor 3, and each replica placed in different zones (us-west-2a, us-west-2b, us-west-2c) in the us-west-2 (Oregon) region of AWS. This can be done by running the following:

  1. $ ./bin/yb-ctl create --placement_info "aws.us-west-2.us-west-2a,aws.us-west-2.us-west-2b,aws.us-west-2.us-west-2c"

In this deployment, the YB Masters are each placed in a separate zone to allow them to survive the loss of a zone. You can view the masters on the dashboard.

Multi-zone universe masters

You can view the tablet servers on the tablet servers page.

Multi-zone universe tservers

2. Start a workload

Run a simple key-value workload in a separate shell.

  1. $ java -jar java/yb-sample-apps.jar \
  2.     --workload CassandraKeyValue \
  3.     --nodes 127.0.0.1:9042 \
  4.     --num_threads_read 1 \
  5.     --num_threads_write 1

You should now see some read and write load on the tablet servers page.

Multi-zone universe load

3. Add nodes in US East and Tokyo regions

Add new nodes

Add a node in the zone us-east-1a of region us-east-1.

  1. $ ./bin/yb-ctl add_node --placement_info "aws.us-east-1.us-east-1a"

Add another node in the zone ap-northeast-1a of region ap-northeast-1.

  1. $ ./bin/yb-ctl add_node --placement_info "aws.ap-northeast-1.ap-northeast-1a"

At this point, these 2 new nodes are added into the cluster but are not taking any read or write IO. This is because YB Master’s initial placement policy of storing data across the zones in us-west-2 region still applies.

Add node in a new region

Update placement policy

Let us now update the placement policy, instructing the YB-Master to place data in the new regions.

  1. $ ./bin/yb-admin --master_addresses 127.0.0.1:7100,127.0 .0.2:7100,127.0.0.3:7100 \
  2.     modify_placement_info aws.us-west-2.us-west-2a,aws.us-east-1.us-east-1a,aws.ap-northeast-1.ap-northeast-1a 3

You should see that the data as well as the IO gradually moves from the nodes in us-west-2b and us-west-2c to the newly added nodes. The tablet servers page should soon look something like the screenshot below.

Multi region workload

4. Retire old nodes

Start new masters

Next we need to move the YB-Master from the old nodes to the new nodes. In order to do so, first start a new masters on the new nodes.

  1. $ ./bin/yb-ctl add_node --master --placement_info "aws.us-east-1.us-east-1a"
  1. $ ./bin/yb-ctl add_node --master --placement_info "aws.ap-northeast-1.ap-northeast-1a"

Add master

Remove old masters

Remove the old masters from the masters Raft group. Assuming nodes with IPs 127.0.0.2 and 127.0.0.3 were the two old nodes, run the following commands.

  1. $ ./bin/yb-admin --master_addresses 127.0.0.1:7100,127.0.0.2:7100,127.0.0.3:7100,127.0.0.4:7100,127.0.0.5:7100 change_master_config REMOVE_SERVER 127.0.0.2 7100
  1. $ ./bin/yb-admin --master_addresses 127.0.0.1:7100,127.0.0.3:7100,127.0.0.4:7100,127.0.0.5:7100 change_master_config REMOVE_SERVER 127.0.0.3 7100

Add master

Remove old nodes

Now it’s safe to remove the old nodes.

  1. $ ./bin/yb-ctl remove_node 2
  1. $ ./bin/yb-ctl remove_node 3

Add master

5. Clean up (optional)

Optionally, you can shutdown the local cluster created in Step 1.

  1. $ ./bin/yb-ctl destroy

1. Create a multi-zone universe in US West

If you have a previously running local universe, destroy it using the following.

  1. $ ./bin/yb-ctl destroy

Start a new local universe with replication factor 3, and each replica placed in different zones (us-west-2a, us-west-2b, us-west-2c) in the us-west-2 (Oregon) region of AWS. This can be done by running the following:

  1. $ ./bin/yb-ctl create --placement_info "aws.us-west-2.us-west-2a,aws.us-west-2.us-west-2b,aws.us-west-2.us-west-2c"

In this deployment, the YB Masters are each placed in a separate zone to allow them to survive the loss of a zone. You can view the masters on the dashboard.

Multi-zone universe masters

You can view the tablet servers on the tablet servers page.

Multi-zone universe tservers

2. Start a workload

Run a simple key-value workload in a separate shell.

  1. $ java -jar java/yb-sample-apps.jar \
  2.     --workload CassandraKeyValue \
  3.     --nodes 127.0.0.1:9042 \
  4.     --num_threads_read 1 \
  5.     --num_threads_write 1

You should now see some read and write load on the tablet servers page.

Multi-zone universe load

3. Add nodes in US East and Tokyo regions

Add new nodes

Add a node in the zone us-east-1a of region us-east-1.

  1. $ ./bin/yb-ctl add_node --placement_info "aws.us-east-1.us-east-1a"

Add another node in the zone ap-northeast-1a of region ap-northeast-1.

  1. $ ./bin/yb-ctl add_node --placement_info "aws.ap-northeast-1.ap-northeast-1a"

At this point, these 2 new nodes are added into the cluster but are not taking any read or write IO. This is because YB Master’s initial placement policy of storing data across the zones in us-west-2 region still applies.

Add node in a new region

Update placement policy

Let us now update the placement policy, instructing the YB-Master to place data in the new regions.

  1. $ ./bin/yb-admin --master_addresses 127.0.0.1:7100,127.0 .0.2:7100,127.0.0.3:7100 \
  2.     modify_placement_info aws.us-west-2.us-west-2a,aws.us-east-1.us-east-1a,aws.ap-northeast-1.ap-northeast-1a 3

You should see that the data as well as the IO gradually moves from the nodes in us-west-2b and us-west-2c to the newly added nodes. The tablet servers page should soon look something like the screenshot below.

Multi region workload

4. Retire old nodes

Start new masters

Next we need to move the YB-Master from the old nodes to the new nodes. In order to do so, first start a new masters on the new nodes.

  1. $ ./bin/yb-ctl add_node --master --placement_info "aws.us-east-1.us-east-1a"
  1. $ ./bin/yb-ctl add_node --master --placement_info "aws.ap-northeast-1.ap-northeast-1a"

Add master

Remove old masters

Remove the old masters from the masters Raft group. Assuming nodes with IPs 127.0.0.2 and 127.0.0.3 were the two old nodes, run the following commands.

  1. $ ./bin/yb-admin --master_addresses 127.0.0.1:7100,127.0.0.2:7100,127.0.0.3:7100,127.0.0.4:7100,127.0.0.5:7100 change_master_config REMOVE_SERVER 127.0.0.2 7100
  1. $ ./bin/yb-admin --master_addresses 127.0.0.1:7100,127.0.0.3:7100,127.0.0.4:7100,127.0.0.5:7100 change_master_config REMOVE_SERVER 127.0.0.3 7100

Add master

Remove old nodes

Now it’s safe to remove the old nodes.

  1. $ ./bin/yb-ctl remove_node 2
  1. $ ./bin/yb-ctl remove_node 3

Add master

5. Clean up (optional)

Optionally, you can shutdown the local cluster created in Step 1.

  1. $ ./bin/yb-ctl destroy