Create auto-follow pattern API

Create auto-follow pattern API

New API reference

For the most up-to-date API details, refer to Cross-cluster replication APIs.

Creates a cross-cluster replication auto-follow pattern.

Request

  1. resp = client.ccr.put_auto_follow_pattern(
  2.     name="<auto_follow_pattern_name>",
  3.     remote_cluster="<remote_cluster>",
  4.     leader_index_patterns=[
  5.         "<leader_index_pattern>"
  6.     ],
  7.     leader_index_exclusion_patterns=[
  8.         "<leader_index_exclusion_pattern>"
  9.     ],
  10.     follow_index_pattern="<follow_index_pattern>",
  11. )
  12. print(resp)
  1. const response = await client.ccr.putAutoFollowPattern({
  2.   name: "<auto_follow_pattern_name>",
  3.   remote_cluster: "<remote_cluster>",
  4.   leader_index_patterns: ["<leader_index_pattern>"],
  5.   leader_index_exclusion_patterns: ["<leader_index_exclusion_pattern>"],
  6.   follow_index_pattern: "<follow_index_pattern>",
  7. });
  8. console.log(response);
  1. PUT /_ccr/auto_follow/<auto_follow_pattern_name>
  2. {
  3.   "remote_cluster" : "<remote_cluster>",
  4.   "leader_index_patterns" :
  5.   [
  6.     "<leader_index_pattern>"
  7.   ],
  8.   "leader_index_exclusion_patterns":
  9.   [
  10.     "<leader_index_exclusion_pattern>"
  11.   ],
  12.   "follow_index_pattern" : "<follow_index_pattern>"
  13. }

Prerequisites

  • If the Elasticsearch security features are enabled, you must have read and monitor index privileges for the leader index patterns. You must also have manage_ccr cluster privileges on the cluster that contains the follower index. For more information, see Security privileges.

Description

This API creates a new named collection of auto-follow patterns against the remote cluster specified in the request body. Newly created indices on the remote cluster matching any of the specified patterns will be automatically configured as follower indices. Indices on the remote cluster that were created before the auto-follow pattern is created won’t be auto-followed even if they match the pattern.

This API can also be used to update existing auto-follow patterns. Note that follower indices that were configured automatically before updating an auto-follow pattern will remain unchanged even if they don’t match against the new patterns.

Path parameters

<auto_follow_pattern_name>

(Required, string) The name of the collection of auto-follow patterns.

Query parameters

master_timeout

(Optional, time units) Period to wait for the master node. If the master node is not available before the timeout expires, the request fails and returns an error. Defaults to 30s. Can also be set to -1 to indicate that the request should never timeout.

Request body

remote_cluster

(Required, string) The remote cluster containing the leader indices to match against.

leader_index_patterns

(Optional, array) An array of simple index patterns to match against indices in the remote cluster specified by the remote_cluster field.

leader_index_exclusion_patterns

(Optional, array) An array of simple index patterns that can be used to exclude indices from being auto-followed. Indices in the remote cluster whose names are matching one or more leader_index_patterns and one or more leader_index_exclusion_patterns won’t be followed.

follow_index_pattern

(Optional, string) The name of follower index. The template {{leader_index}} can be used to derive the name of the follower index from the name of the leader index. When following a data stream, the follow_index_pattern will be used for renaming not only the leader index, but also the data stream containing the leader index. For example, a data stream called logs-mysql-default with a backing index of .ds-logs-mysql-default-2022-01-01-000001 and a follow_index_pattern of {{leader_index}}_copy will replicate the data stream as logs-mysql-default_copy and the backing index as .ds-logs-mysql-default_copy-2022-01-01-000001.

settings

(object) Settings to override from the leader index. Note that certain settings can not be overrode (e.g., index.number_of_shards).

max_read_request_operation_count

(integer) The maximum number of operations to pull per read from the remote cluster.

max_outstanding_read_requests

(long) The maximum number of outstanding reads requests from the remote cluster.

max_read_request_size

(byte value) The maximum size in bytes of per read of a batch of operations pulled from the remote cluster.

max_write_request_operation_count

(integer) The maximum number of operations per bulk write request executed on the follower.

max_write_request_size

(byte value) The maximum total bytes of operations per bulk write request executed on the follower.

max_outstanding_write_requests

(integer) The maximum number of outstanding write requests on the follower.

max_write_buffer_count

(integer) The maximum number of operations that can be queued for writing. When this limit is reached, reads from the remote cluster will be deferred until the number of queued operations goes below the limit.

max_write_buffer_size

(byte value) The maximum total bytes of operations that can be queued for writing. When this limit is reached, reads from the remote cluster will be deferred until the total bytes of queued operations goes below the limit.

max_retry_delay

(time value) The maximum time to wait before retrying an operation that failed exceptionally. An exponential backoff strategy is employed when retrying.

read_poll_timeout

(time value) The maximum time to wait for new operations on the remote cluster when the follower index is synchronized with the leader index. When the timeout has elapsed, the poll for operations will return to the follower so that it can update some statistics. Then the follower will immediately attempt to read from the leader again.

Default values

The following output from the follow info api describes all the default values for the above described index follow request parameters:

  1. {
  2.   "follower_indices" : [
  3.     {
  4.       "parameters" : {
  5.         "max_read_request_operation_count" : 5120,
  6.         "max_read_request_size" : "32mb",
  7.         "max_outstanding_read_requests" : 12,
  8.         "max_write_request_operation_count" : 5120,
  9.         "max_write_request_size" : "9223372036854775807b",
  10.         "max_outstanding_write_requests" : 9,
  11.         "max_write_buffer_count" : 2147483647,
  12.         "max_write_buffer_size" : "512mb",
  13.         "max_retry_delay" : "500ms",
  14.         "read_poll_timeout" : "1m"
  15.       }
  16.     }
  17.   ]
  18. }

Examples

This example creates an auto-follow pattern named my_auto_follow_pattern:

  1. resp = client.ccr.put_auto_follow_pattern(
  2.     name="my_auto_follow_pattern",
  3.     remote_cluster="remote_cluster",
  4.     leader_index_patterns=[
  5.         "leader_index*"
  6.     ],
  7.     follow_index_pattern="{{leader_index}}-follower",
  8.     settings={
  9.         "index.number_of_replicas": 0
  10.     },
  11.     max_read_request_operation_count=1024,
  12.     max_outstanding_read_requests=16,
  13.     max_read_request_size="1024k",
  14.     max_write_request_operation_count=32768,
  15.     max_write_request_size="16k",
  16.     max_outstanding_write_requests=8,
  17.     max_write_buffer_count=512,
  18.     max_write_buffer_size="512k",
  19.     max_retry_delay="10s",
  20.     read_poll_timeout="30s",
  21. )
  22. print(resp)
  1. const response = await client.ccr.putAutoFollowPattern({
  2.   name: "my_auto_follow_pattern",
  3.   remote_cluster: "remote_cluster",
  4.   leader_index_patterns: ["leader_index*"],
  5.   follow_index_pattern: "{{leader_index}}-follower",
  6.   settings: {
  7.     "index.number_of_replicas": 0,
  8.   },
  9.   max_read_request_operation_count: 1024,
  10.   max_outstanding_read_requests: 16,
  11.   max_read_request_size: "1024k",
  12.   max_write_request_operation_count: 32768,
  13.   max_write_request_size: "16k",
  14.   max_outstanding_write_requests: 8,
  15.   max_write_buffer_count: 512,
  16.   max_write_buffer_size: "512k",
  17.   max_retry_delay: "10s",
  18.   read_poll_timeout: "30s",
  19. });
  20. console.log(response);
  1. PUT /_ccr/auto_follow/my_auto_follow_pattern
  2. {
  3.   "remote_cluster" : "remote_cluster",
  4.   "leader_index_patterns" :
  5.   [
  6.     "leader_index*"
  7.   ],
  8.   "follow_index_pattern" : "{{leader_index}}-follower",
  9.   "settings": {
  10.     "index.number_of_replicas": 0
  11.   },
  12.   "max_read_request_operation_count" : 1024,
  13.   "max_outstanding_read_requests" : 16,
  14.   "max_read_request_size" : "1024k",
  15.   "max_write_request_operation_count" : 32768,
  16.   "max_write_request_size" : "16k",
  17.   "max_outstanding_write_requests" : 8,
  18.   "max_write_buffer_count" : 512,
  19.   "max_write_buffer_size" : "512k",
  20.   "max_retry_delay" : "10s",
  21.   "read_poll_timeout" : "30s"
  22. }

The API returns the following result:

  1. {
  2.   "acknowledged" : true
  3. }