Configure Replication Zones

Replication zones give you the power to control what data goes where in your CockroachDB cluster. Specifically, they are used to control the number and location of replicas for data belonging to the following objects:

• Databases
• Tables
• Rows (enterprise-only)
• Indexes (enterprise-only)

• All data in the cluster, including internal system data (via the default replication zone)
  For each of the above objects you can control:

• How many copies of each range to spread through the cluster.

• Which constraints are applied to which data, e.g., "table X's data can only be stored in the German datacenters".
• The maximum size of ranges (how big ranges get before they are split).
• How long old data is kept before being garbage collected.
• Where you would like the leaseholders for certain ranges to be located, e.g., "for ranges that are already constrained to have at least one replica in region=us-west, also try to put their leaseholders in region=us-west".
  This page explains how replication zones work and how to use the CONFIGURE ZONE statement to manage them.

Note:

Currently, only members of the admin role can configure replication zones. By default, the root user belongs to the admin role.

Overview

Every range in the cluster is part of a replication zone. Each range's zone configuration is taken into account as ranges are rebalanced across the cluster to ensure that any constraints are honored.

When a cluster starts, there are two categories of replication zone:

• Pre-configured replication zones that apply to internal system data.
• A single default replication zone that applies to the rest of the cluster.
  You can adjust these pre-configured zones as well as add zones for individual databases, tables, rows, and secondary indexes as needed. Note that adding zones for rows and secondary indexes is enterprise-only.

For example, you might rely on the default zone to spread most of a cluster's data across all of your datacenters, but create a custom replication zone for a specific database to make sure its data is only stored in certain datacenters and/or geographies.

Replication zone levels

There are five replication zone levels for table data in a cluster, listed from least to most granular:

For system data

In addition, CockroachDB stores internal system data in what are called system ranges. There are two replication zone levels for this internal system data, listed from least to most granular:

Level priorities

When replicating data, whether table or system, CockroachDB always uses the most granular replication zone available. For example, for a piece of user data:

• If there's a replication zone for the row, CockroachDB uses it.
• If there's no applicable row replication zone and the row is from a secondary index, CockroachDB uses the secondary index replication zone.
• If the row isn't from a secondary index or there is no applicable secondary index replication zone, CockroachDB uses the table replication zone.
• If there's no applicable table replication zone, CockroachDB uses the database replication zone.
• If there's no applicable database replication zone, CockroachDB uses the .default cluster-wide replication zone.

Manage replication zones

Use the CONFIGURE ZONE statement to add, modify, reset, and remove replication zones.

Replication zone variables

Use the ALTER … CONFIGURE ZONEstatement to set a replication zone:

> ALTER TABLE t CONFIGURE ZONE USING range_min_bytes = 0, range_max_bytes = 90000, gc.ttlseconds = 89999, num_replicas = 5, constraints = '[-region=west]';

Note:

If a value is not set, new zone configurations will inherit their values from their parent zone (e.g., a partition zone inherits from the table zone), which is not necessarily .default.

If a variable is set to COPY FROM PARENT (e.g., range_max_bytes = COPY FROM PARENT), the variable will copy its value from its parent replication zone. The COPY FROM PARENT value is a convenient shortcut to use so you do not have to look up the parent's current value. For example, the range_max_bytes and range_min_bytes variables must be set together, so when editing one value, you can use COPY FROM PARENT for the other. Note that if the variable in the parent replication zone is changed after the child replication zone is copied, the change will not be reflected in the child zone.

Replication constraints

The location of replicas, both when they are first added and when they are rebalanced to maintain cluster equilibrium, is based on the interplay between descriptive attributes assigned to nodes and constraints set in zone configurations.

Tip:
For demonstrations of how to set node attributes and replication constraints in different scenarios, see Scenario-based Examples below.

Descriptive attributes assigned to nodes

When starting a node with the cockroach start command, you can assign the following types of descriptive attributes:

Types of constraints

The node-level and store-level descriptive attributes mentioned above can be used as the following types of constraints in replication zones to influence the location of replicas. However, note the following general guidance:

• When locality is the only consideration for replication, it's recommended to set locality on nodes without specifying any constraints in zone configurations. In the absence of constraints, CockroachDB attempts to spread replicas evenly across the cluster based on locality.
• Required and prohibited constraints are useful in special situations where, for example, data must or must not be stored in a specific country or on a specific type of machine.

  Constraint Type	Description	Syntax
  Required	When placing replicas, the cluster will consider only nodes/stores with matching attributes or localities. When there are no matching nodes/stores, new replicas will not be added.	+ssd
  Prohibited	When placing replicas, the cluster will ignore nodes/stores with matching attributes or localities. When there are no alternate nodes/stores, new replicas will not be added.	-ssd

Scope of constraints

Constraints can be specified such that they apply to all replicas in a zone or such that different constraints apply to different replicas, meaning you can effectively pick the exact location of each replica.

Node/replica recommendations

See Cluster Topography recommendations for production deployments.

View replication zones

Use the SHOW ZONE CONFIGURATIONS statement to view details about existing replication zones.

Basic examples

These examples focus on the basic approach and syntax for working with zone configuration. For examples demonstrating how to use constraints, see Scenario-based examples.

For more examples, see CONFIGURE ZONE and SHOW ZONE CONFIGURATIONS.

View all replication zones

> SHOW ALL ZONE CONFIGURATIONS;

   zone_name  |                     config_sql
+-------------+-----------------------------------------------------+
  .default    | ALTER RANGE default CONFIGURE ZONE USING
              |     range_min_bytes = 1048576,
              |     range_max_bytes = 67108864,
              |     gc.ttlseconds = 90000,
              |     num_replicas = 3,
              |     constraints = '[]',
              |     lease_preferences = '[]'
  system      | ALTER DATABASE system CONFIGURE ZONE USING
              |     range_min_bytes = 1048576,
              |     range_max_bytes = 67108864,
              |     gc.ttlseconds = 90000,
              |     num_replicas = 5,
              |     constraints = '[]',
              |     lease_preferences = '[]'
  system.jobs | ALTER TABLE system.public.jobs CONFIGURE ZONE USING
              |     range_min_bytes = 1048576,
              |     range_max_bytes = 67108864,
              |     gc.ttlseconds = 600,
              |     num_replicas = 5,
              |     constraints = '[]',
              |     lease_preferences = '[]'
  .meta       | ALTER RANGE meta CONFIGURE ZONE USING
              |     range_min_bytes = 1048576,
              |     range_max_bytes = 67108864,
              |     gc.ttlseconds = 3600,
              |     num_replicas = 5,
              |     constraints = '[]',
              |     lease_preferences = '[]'
  .system     | ALTER RANGE system CONFIGURE ZONE USING
              |     range_min_bytes = 1048576,
              |     range_max_bytes = 67108864,
              |     gc.ttlseconds = 90000,
              |     num_replicas = 5,
              |     constraints = '[]',
              |     lease_preferences = '[]'
  .liveness   | ALTER RANGE liveness CONFIGURE ZONE USING
              |     range_min_bytes = 1048576,
              |     range_max_bytes = 67108864,
              |     gc.ttlseconds = 600,
              |     num_replicas = 5,
              |     constraints = '[]',
              |     lease_preferences = '[]'
(6 rows)

For more information, see SHOW ZONE CONFIGURATIONS.

View the default replication zone

> SHOW ZONE CONFIGURATION FOR RANGE default;

  zone_name |                config_sql
+-----------+------------------------------------------+
  .default  | ALTER RANGE default CONFIGURE ZONE USING
            |     range_min_bytes = 1048576,
            |     range_max_bytes = 67108864,
            |     gc.ttlseconds = 90000,
            |     num_replicas = 3,
            |     constraints = '[]',
            |     lease_preferences = '[]'
(1 row)

For more information, see SHOW ZONE CONFIGURATIONS.

Edit the default replication zone

To edit the default replication zone, use the ALTER RANGE … CONFIGURE ZONE statement to define the values you want to change (other values will remain the same):

> ALTER RANGE default CONFIGURE ZONE USING num_replicas = 5, gc.ttlseconds = 100000;

CONFIGURE ZONE 1

> SHOW ZONE CONFIGURATION FOR RANGE default;

  zone_name |                config_sql
+-----------+------------------------------------------+
  .default  | ALTER RANGE default CONFIGURE ZONE USING
            |     range_min_bytes = 1048576,
            |     range_max_bytes = 67108864,
            |     gc.ttlseconds = 100000,
            |     num_replicas = 5,
            |     constraints = '[]',
            |     lease_preferences = '[]'
(1 row)

For more information, see CONFIGURE ZONE.

Create a replication zone for a system range

In addition to the databases and tables that are visible via the SQL interface, CockroachDB stores internal data in what are called system ranges. CockroachDB comes with pre-configured replication zones for some of these ranges:

Warning:

Use caution when editing replication zones for system ranges, as they could cause some (or all) parts of your cluster to stop working.

To control replication for one of the above sets of system ranges, use the ALTER RANGE … CONFIGURE ZONE statement to define the values you want to change (other values will not be affected):

> ALTER RANGE meta CONFIGURE ZONE USING num_replicas = 7;

CONFIGURE ZONE 1

> SHOW ZONE CONFIGURATION FOR RANGE meta;

  zone_name |              config_sql
+-----------+---------------------------------------+
  .meta     | ALTER RANGE meta CONFIGURE ZONE USING
            |     range_min_bytes = 1048576,
            |     range_max_bytes = 67108864,
            |     gc.ttlseconds = 3600,
            |     num_replicas = 7,
            |     constraints = '[]',
            |     lease_preferences = '[]'
(1 row)

For more information, see CONFIGURE ZONE.

Create a replication zone for a database

To control replication for a specific database, use the ALTER DATABASE … CONFIGURE ZONE statement to define the values you want to change (other values will not be affected):

> ALTER DATABASE test CONFIGURE ZONE USING num_replicas = 5, gc.ttlseconds = 100000;

CONFIGURE ZONE 1

> SHOW ZONE CONFIGURATION FOR DATABASE test;

  zone_name |                config_sql
+-----------+------------------------------------------+
  test      | ALTER DATABASE test CONFIGURE ZONE USING
            |     range_min_bytes = 1048576,
            |     range_max_bytes = 67108864,
            |     gc.ttlseconds = 100000,
            |     num_replicas = 5,
            |     constraints = '[]',
            |     lease_preferences = '[]'
(1 row)

For more information, see CONFIGURE ZONE.

Create a replication zone for a table

To control replication for a specific table, use the ALTER TABLE … CONFIGURE ZONE statement to define the values you want to change (other values will not be affected):

> ALTER TABLE customers CONFIGURE ZONE USING num_replicas = 5, gc.ttlseconds = 100000;

CONFIGURE ZONE 1

> SHOW ZONE CONFIGURATION FOR TABLE customers;

    zone_name    |                 config_sql
+----------------+--------------------------------------------+
  test.customers | ALTER TABLE customers CONFIGURE ZONE USING
                 |     range_min_bytes = 1048576,
                 |     range_max_bytes = 67108864,
                 |     gc.ttlseconds = 100000,
                 |     num_replicas = 5,
                 |     constraints = '[]',
                 |     lease_preferences = '[]'
(1 row)

For more information, see CONFIGURE ZONE.

Create a replication zone for a secondary index

Tip:

New in v19.1: The Cost-based Optimizer can take advantage of replication zones for secondary indexes when optimizing queries. For more information, see Cost-based optimizer - preferring the nearest index.

Note:

This is an enterprise-only feature.

The secondary indexes on a table will automatically use the replication zone for the table. However, with an enterprise license, you can add distinct replication zones for secondary indexes.

To control replication for a specific secondary index, use the ALTER INDEX … CONFIGURE ZONE statement to define the values you want to change (other values will not be affected).

Tip:

To get the name of a secondary index, which you need for the CONFIGURE ZONE statement, use the SHOW INDEX or SHOW CREATE TABLE statements.

> ALTER INDEX tpch.frequent_customers CONFIGURE ZONE USING num_replicas = 5, gc.ttlseconds = 100000;

CONFIGURE ZONE 1

> SHOW ZONE CONFIGURATION FOR INDEX tpch.customer@frequent_customers;

             zone_name             |                                config_sql
+----------------------------------+--------------------------------------------------------------------------+
  tpch.customer@frequent_customers | ALTER INDEX tpch.public.customer@frequent_customers CONFIGURE ZONE USING
                                   |     range_min_bytes = 1048576,
                                   |     range_max_bytes = 67108864,
                                   |     gc.ttlseconds = 100000,
                                   |     num_replicas = 5,
                                   |     constraints = '[]',
                                   |     lease_preferences = '[]'
(1 row)

For more information, see CONFIGURE ZONE.

Create a replication zone for a table or secondary index partition

Note:

This is an enterprise-only feature.

To control replication for table partitions, use the ALTER PARTITION … CONFIGURE ZONE statement to define the values you want to change (other values will not be affected):

> ALTER PARTITION north_america OF TABLE customers CONFIGURE ZONE USING num_replicas = 5, constraints = '[-region=EU]';

CONFIGURE ZONE 1

> SHOW ZONE CONFIGURATION FOR PARTITION north_america OF TABLE customers;

           zone_name           |                                  config_sql
+------------------------------+-------------------------------------------------------------------------------+
  test.customers.north_america | ALTER PARTITION north_america OF INDEX customers@primary CONFIGURE ZONE USING
                               |     range_min_bytes = 1048576,
                               |     range_max_bytes = 67108864,
                               |     gc.ttlseconds = 100000,
                               |     num_replicas = 5,
                               |     constraints = '[-region=EU]',
                               |     lease_preferences = '[]'
(1 row)

Tip:

Since the syntax is the same for defining a replication zone for a table or index partition (e.g., database.table.partition), give partitions names that communicate what they are partitioning, e.g., north_america_table vs north_america_idx1.

For more information, see CONFIGURE ZONE.

Reset a replication zone

> ALTER TABLE t CONFIGURE ZONE USING DEFAULT;

CONFIGURE ZONE 1

For more information, see CONFIGURE ZONE.

Remove a replication zone

> ALTER TABLE t CONFIGURE ZONE DISCARD;

CONFIGURE ZONE 1

For more information, see CONFIGURE ZONE.

Constrain leaseholders to specific datacenters

In addition to constraining replicas to specific datacenters, you may also specify preferences for where the range's leaseholders should be placed. This can result in increased performance in some scenarios.

The ALTER TABLE … CONFIGURE ZONE statement below requires that the cluster try to place the ranges' leaseholders in zone us-east-1b; if that is not possible, it will try to place them in zone us-east-1a.

For more information about how the lease_preferences field works, see its description in the Replication zone variables section.

> ALTER TABLE kv CONFIGURE ZONE USING num_replicas = 3, constraints = '{"+zone=us-east-1a": 1, "+zone=us-east-1b": 1}', lease_preferences = '[[+zone=us-east-1b], [+zone=us-east-1a]]';

CONFIGURE ZONE 1

> SHOW ZONE CONFIGURATION FOR TABLE kv;

 zone_name |                               config_sql
-----------+------------------------------------------------------------------------
 test.kv   | ALTER TABLE kv CONFIGURE ZONE USING                                   +
           |         range_min_bytes = 1048576,                                    +
           |         range_max_bytes = 67108864,                                   +
           |         gc.ttlseconds = 90000,                                        +
           |         num_replicas = 3,                                             +
           |         constraints = '{+zone=us-east-1a: 1, +zone=us-east-1b: 1}',   +
           |         lease_preferences = '[[+zone=us-east-1b], [+zone=us-east-1a]]'
(1 row)

For more information, see CONFIGURE ZONE.

Scenario-based examples

Even replication across datacenters

Scenario:

• You have 6 nodes across 3 datacenters, 2 nodes in each datacenter.
• You want data replicated 3 times, with replicas balanced evenly across all three datacenters.
  Approach:

Start each node with its datacenter location specified in the —locality flag:

# Start the two nodes in datacenter 1:
$ cockroach start --insecure --advertise-addr=<node1 hostname> --locality=datacenter=us-1 \
--join=<node1 hostname>,<node3 hostname>,<node5 hostname>
$ cockroach start --insecure --advertise-addr=<node2 hostname> --locality=datacenter=us-1 \
--join=<node1 hostname>,<node3 hostname>,<node5 hostname>
# Start the two nodes in datacenter 2:
$ cockroach start --insecure --advertise-addr=<node3 hostname> --locality=datacenter=us-2 \
--join=<node1 hostname>,<node3 hostname>,<node5 hostname>
$ cockroach start --insecure --advertise-addr=<node4 hostname> --locality=datacenter=us-2 \
--join=<node1 hostname>,<node3 hostname>,<node5 hostname>
# Start the two nodes in datacenter 3:
$ cockroach start --insecure --advertise-addr=<node5 hostname> --locality=datacenter=us-3 \
--join=<node1 hostname>,<node3 hostname>,<node5 hostname>
$ cockroach start --insecure --advertise-addr=<node6 hostname> --locality=datacenter=us-3 \
--join=<node1 hostname>,<node3 hostname>,<node5 hostname>
# Initialize the cluster:
$ cockroach init --insecure --host=<any node hostname>

There's no need to make zone configuration changes; by default, the cluster is configured to replicate data three times, and even without explicit constraints, the cluster will aim to diversify replicas across node localities.

Per-replica constraints to specific datacenters

Scenario:

• You have 5 nodes across 5 datacenters in 3 regions, 1 node in each datacenter.

• You want data replicated 3 times, with a quorum of replicas for a database holding West Coast data centered on the West Coast and a database for nation-wide data replicated across the entire country.
  Approach:

• Start each node with its region and datacenter location specified in the —locality flag:

# Start the four nodes:
$ cockroach start --insecure --advertise-addr=<node1 hostname> --locality=region=us-west1,datacenter=us-west1-a \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>
$ cockroach start --insecure --advertise-addr=<node2 hostname> --locality=region=us-west1,datacenter=us-west1-b \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>
$ cockroach start --insecure --advertise-addr=<node3 hostname> --locality=region=us-central1,datacenter=us-central1-a \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>
$ cockroach start --insecure --advertise-addr=<node4 hostname> --locality=region=us-east1,datacenter=us-east1-a \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>
$ cockroach start --insecure --advertise-addr=<node5 hostname> --locality=region=us-east1,datacenter=us-east1-b \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>
# Initialize the cluster:
$ cockroach init --insecure --host=<any node hostname>

• On any node, open the built-in SQL client:

$ cockroach sql --insecure

• Create the database for the West Coast application:

> CREATE DATABASE west_app_db;

• Configure a replication zone for the database:

> ALTER DATABASE west_app_db \
CONFIGURE ZONE USING constraints = '{"+region=us-west1": 2, "+region=us-central1": 1}', num_replicas = 3;

CONFIGURE ZONE 1

• View the replication zone:

> SHOW ZONE CONFIGURATION FOR DATABASE west_app_db;

   zone_name  |                   config_sql
+-------------+--------------------------------------------------------------------+
  west_app_db | ALTER DATABASE west_app_db CONFIGURE ZONE USING
              |     range_min_bytes = 1048576,
              |     range_max_bytes = 67108864,
              |     gc.ttlseconds = 90000,
              |     num_replicas = 3,
              |     constraints = '{+region=us-west1: 2, +region=us-central1: 1}',
              |     lease_preferences = '[]'
(1 row)

Two of the database's three replicas will be put in region=us-west1 and its remaining replica will be put in region=us-central1. This gives the application the resilience to survive the total failure of any one datacenter while providing low-latency reads and writes on the West Coast because a quorum of replicas are located there.

• No configuration is needed for the nation-wide database. The cluster is configured to replicate data 3 times and spread them as widely as possible by default. Because the first key-value pair specified in each node's locality is considered the most significant part of each node's locality, spreading data as widely as possible means putting one replica in each of the three different regions.

Multiple applications writing to different databases

Scenario:

• You have 2 independent applications connected to the same CockroachDB cluster, each application using a distinct database.
• You have 6 nodes across 2 datacenters, 3 nodes in each datacenter.
• You want the data for application 1 to be replicated 5 times, with replicas evenly balanced across both datacenters.

• You want the data for application 2 to be replicated 3 times, with all replicas in a single datacenter.
  Approach:

• Start each node with its datacenter location specified in the —locality flag:

# Start the three nodes in datacenter 1:
$ cockroach start --insecure --advertise-addr=<node1 hostname> --locality=datacenter=us-1 \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>,<node6 hostname>
$ cockroach start --insecure --advertise-addr=<node2 hostname> --locality=datacenter=us-1 \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>,<node6 hostname>
$ cockroach start --insecure --advertise-addr=<node3 hostname> --locality=datacenter=us-1 \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>,<node6 hostname>
# Start the three nodes in datacenter 2:
$ cockroach start --insecure --advertise-addr=<node4 hostname> --locality=datacenter=us-2 \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>,<node6 hostname>
$ cockroach start --insecure --advertise-addr=<node5 hostname> --locality=datacenter=us-2 \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>,<node6 hostname>
$ cockroach start --insecure --advertise-addr=<node6 hostname> --locality=datacenter=us-2 \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>,<node6 hostname>
# Initialize the cluster:
$ cockroach init --insecure --host=<any node hostname>

• On any node, open the built-in SQL client:

$ cockroach sql --insecure

• Create the database for application 1:

> CREATE DATABASE app1_db;

• Configure a replication zone for the database used by application 1:

> ALTER DATABASE app1_db CONFIGURE ZONE USING num_replicas = 5;

CONFIGURE ZONE 1

• View the replication zone:

> SHOW ZONE CONFIGURATION FOR DATABASE app1_db;

    zone_name  |                config_sql
+--------------+---------------------------------------------+
     app1_db   | ALTER DATABASE app1_db CONFIGURE ZONE USING
               |     range_min_bytes = 1048576,
               |     range_max_bytes = 67108864,
               |     gc.ttlseconds = 90000,
               |     num_replicas = 5,
               |     constraints = '[]',
               |     lease_preferences = '[]'
(1 row)

Nothing else is necessary for application 1's data. Since all nodes specify their datacenter locality, the cluster will aim to balance the data in the database used by application 1 between datacenters 1 and 2.

• Still in the SQL client, create a database for application 2:

> CREATE DATABASE app2_db;

• Configure a replication zone for the database used by application 2:

> ALTER DATABASE app2_db CONFIGURE ZONE USING constraints = '[+datacenter=us-2]';

• View the replication zone:

> SHOW ZONE CONFIGURATION FOR DATABASE app2_db;

    zone_name  |                config_sql
+--------------+---------------------------------------------+
     app2_db   | ALTER DATABASE app2_db CONFIGURE ZONE USING
               |     range_min_bytes = 1048576,
               |     range_max_bytes = 67108864,
               |     gc.ttlseconds = 90000,
               |     num_replicas = 3,
               |     constraints = '[+datacenter=us-2]',
               |     lease_preferences = '[]'
(1 row)

The required constraint will force application 2's data to be replicated only within the us-2 datacenter.

Stricter replication for a table and its secondary indexes

Scenario:

• You have 7 nodes, 5 with SSD drives and 2 with HDD drives.
• You want data replicated 3 times by default.

• Speed and availability are important for a specific table and its indexes, which are queried very frequently, however, so you want the data in the table and secondary indexes to be replicated 5 times, preferably on nodes with SSD drives.
  Approach:

• Start each node with ssd or hdd specified as store attributes:

# Start the 5 nodes with SSD storage:
$ cockroach start --insecure --advertise-addr=<node1 hostname> --store=path=node1,attrs=ssd \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>
$ cockroach start --insecure --advertise-addr=<node2 hostname> --store=path=node2,attrs=ssd \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>
$ cockroach start --insecure --advertise-addr=<node3 hostname> --store=path=node3,attrs=ssd \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>
$ cockroach start --insecure --advertise-addr=<node4 hostname> --store=path=node4,attrs=ssd \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>
$ cockroach start --insecure --advertise-addr=<node5 hostname> --store=path=node5,attrs=ssd \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>
# Start the 2 nodes with HDD storage:
$ cockroach start --insecure --advertise-addr=<node6 hostname> --store=path=node6,attrs=hdd \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>
$ cockroach start --insecure --advertise-addr=<node7 hostname> --store=path=node7,attrs=hdd \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>
# Initialize the cluster:
$ cockroach init --insecure --host=<any node hostname>

• On any node, open the built-in SQL client:

$ cockroach sql --insecure

• Create a database and table:

> CREATE DATABASE db;

> CREATE TABLE db.important_table;

• Configure a replication zone for the table that must be replicated more strictly:

> ALTER TABLE db.important_table CONFIGURE ZONE USING num_replicas = 5, constraints = '[+ssd]'

• View the replication zone:

> SHOW ZONE CONFIGURATION FOR TABLE db.important_table;

          zone_name       |                config_sql
+-------------------------+---------------------------------------------+
     db.important_table   | ALTER DATABASE app2_db CONFIGURE ZONE USING
                          |     range_min_bytes = 1048576,
                          |     range_max_bytes = 67108864,
                          |     gc.ttlseconds = 90000,
                          |     num_replicas = 5,
                          |     constraints = '[+ssd]',
                          |     lease_preferences = '[]'
(1 row)

The secondary indexes on the table will use the table's replication zone, so all data for the table will be replicated 5 times, and the required constraint will place the data on nodes with ssd drives.

Tweaking the replication of system ranges

Scenario:

• You have nodes spread across 7 datacenters.
• You want data replicated 5 times by default.
• For better performance, you want a copy of the meta ranges in all of the datacenters.

• To save disk space, you only want the internal timeseries data replicated 3 times by default.
  Approach:

• Start each node with a different locality attribute:

# Start the nodes:
$ cockroach start --insecure --advertise-addr=<node1 hostname> --locality=datacenter=us-1 \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>   
$ cockroach start --insecure --advertise-addr=<node2 hostname> --locality=datacenter=us-2 \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>
$ cockroach start --insecure --advertise-addr=<node3 hostname> --locality=datacenter=us-3 \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>
$ cockroach start --insecure --advertise-addr=<node4 hostname> --locality=datacenter=us-4 \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>
$ cockroach start --insecure --advertise-addr=<node5 hostname> --locality=datacenter=us-5 \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>
$ cockroach start --insecure --advertise-addr=<node6 hostname> --locality=datacenter=us-6 \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>
$ cockroach start --insecure --advertise-addr=<node7 hostname> --locality=datacenter=us-7 \
--join=<node1 hostname>,<node2 hostname>,<node3 hostname>
# Initialize the cluster:
$ cockroach init --insecure --host=<any node hostname>

• On any node, open the built-in SQL client:

$ cockroach sql --insecure

• Configure the default replication zone:

> ALTER RANGE default CONFIGURE ZONE USING num_replicas = 5;

• View the replication zone:

> SHOW ZONE CONFIGURATION FOR RANGE default;

  zone_name |                config_sql
+-----------+------------------------------------------+
  .default  | ALTER RANGE default CONFIGURE ZONE USING
            |     range_min_bytes = 1048576,
            |     range_max_bytes = 67108864,
            |     gc.ttlseconds = 90000,
            |     num_replicas = 5,
            |     constraints = '[]',
            |     lease_preferences = '[]'
(1 row)

All data in the cluster will be replicated 5 times, including both SQL data and the internal system data.

• Configure the .meta replication zone:

> ALTER RANGE meta CONFIGURE ZONE USING num_replicas = 7;

  zone_name |              config_sql
+-----------+---------------------------------------+
  .meta     | ALTER RANGE meta CONFIGURE ZONE USING
            |     range_min_bytes = 1048576,
            |     range_max_bytes = 67108864,
            |     gc.ttlseconds = 3600,
            |     num_replicas = 7,
            |     constraints = '[]',
            |     lease_preferences = '[]'
(1 row)

The .meta addressing ranges will be replicated such that one copy is in all 7 datacenters, while all other data will be replicated 5 times.

• Configure the .timeseries replication zone:

> ALTER RANGE timeseries CONFIGURE ZONE USING num_replicas = 3;

   zone_name  |                 config_sql
+-------------+---------------------------------------------+
  .timeseries | ALTER RANGE timeseries CONFIGURE ZONE USING
              |     range_min_bytes = 1048576,
              |     range_max_bytes = 67108864,
              |     gc.ttlseconds = 90000,
              |     num_replicas = 3,
              |     constraints = '[]',
              |     lease_preferences = '[]'
(1 row)

The timeseries data will only be replicated 3 times without affecting the configuration of all other data.

See also

• SHOW ZONE CONFIGURATIONS
• CONFIGURE ZONE
• SQL Statements
• Table Partitioning

Was this page helpful?
YesNo