Upgrading clusters using cross-cluster replication

Clusters that are actively using cross-cluster replication require a careful approach to upgrades. The following conditions could cause index following to fail during rolling upgrades:

  • Clusters that have not yet been upgraded will reject new index settings or mapping types that are replicated from an upgraded cluster.
  • Nodes in a cluster that has not been upgraded will reject index files from a node in an upgraded cluster when index following tries to fall back to file-based recovery. This limitation is due to Lucene not being forward compatible.

The approach to running a rolling upgrade on clusters where cross-cluster replication is enabled differs based on uni-directional and bi-directional index following.

Uni-directional index following

In a uni-directional configuration, one cluster contains only leader indices, and the other cluster contains only follower indices that replicate the leader indices.

In this strategy, the cluster with follower indices should be upgraded first and the cluster with leader indices should be upgraded last. Upgrading the clusters in this order ensures that index following can continue during the upgrade without downtime.

You can also use this strategy to upgrade a replication chain. Start by upgrading clusters at the end of the chain and working your way back to the cluster that contains the leader indices.

For example, consider a configuration where Cluster A contains all leader indices. Cluster B follows indices in Cluster A, and Cluster C follows indices in Cluster B.

  1. Cluster A
  2.         ^--Cluster B
  3.                    ^--Cluster C

In this configuration, upgrade the clusters in the following order:

  1. Cluster C
  2. Cluster B
  3. Cluster A

Bi-directional index following

In a bi-directional configuration, each cluster contains both leader and follower indices.

When upgrading clusters in this configuration, pause all index following and pause auto-follow patterns prior to upgrading both clusters.

After upgrading both clusters, resume index following and resume replication of auto-follow patterns.