DevOps

Elasticsearch Data Migration Across Clusters Using Cross-Cluster Replication

9 min read
Share:

Introduction

Managing Elasticsearch clusters in environments is a common challenge for organizations that run search and analytics work that is vital to the business. Whether you are moving to infrastructure, setting up disaster recovery or changing operating systems, keeping Elasticsearch data consistent and minimizing downtime are the top priorities.

The primary objectives were to:

  • Achieve near real-time data replication between clusters
  • Minimize application downtime during migration
  • Ensure data integrity throughout the process

In this blog I will explain:

  • Cross-Cluster Replication (CCR)
  • How CCR works
  • Prerequisites and setup requirements
  • Implementation step, by step
  • Disabling CCR
  • Best practices and Lessons learned

What is Cross-Cluster Replication?

Cross-cluster replication is an Elasticsearch feature that enables you to replicate indices from one cluster (the leader cluster) to another cluster (the follower cluster). We have seen that the follower index is constantly pulling the changes from the leader index to keep the data up to time.

Cross-Cluster Replication is great for:

  • Disaster recovery: Cluster running in different region
  • Data Locality: Move data closer to users in different geographic locations.
  • Migration: Data transfer between clusters during infrastructure upgrade.
  • Centralized reporting: Gather data from several clusters to generate analytics.

How CCR Works

Cross-Cluster Replication works at the shard level. Every follower shard pulls changes from its leader shard. Two main phases exist:

  • Initialization Phase: The follower cluster requests the leader cluster to send all segment files for the index as if it were restoring from a snapshot.
  • Incremental Sync Phase: The follower shard periodically requests read operations from the leader shard starting from the synchronization point. The leader shard responds with operations. The follower replays them locally.

The transaction log is very important. It records every write operation. It serves as the source for synchronization.

Compatibility with Versions

Both clusters must be compatible versions before CCR is set up.

  • Same major version: Any node can talk to another on the same major version.
  • Adjacent major versions: CCR works between the last minor of one major and the next major version. So 7.17 can replicate to and from 8.x, but 7.16 can not talk to 8.0.
  • Newer major versions: A newer follower can replicate from an older leader, but not vise versa. Thus 8.x can follow 7.17 but 7.17 cannot follow 8.x.
  • License: CCR requires a trial, platinum, or enterprise license on both clusters.

If possible, keep both clusters on the same major version. When upgrading major versions or the final minor of the older version, upgrade the leader first.

Prerequisites

  • Two Elasticsearch clusters that have X-Pack enabled.
  • An Elasticsearch license, either trial or platinum.
  • Network connectivity between clusters on the transport port.
  • TLS certificates set up for communication.
  • Adequate storage and compute resources, on the follower cluster.

Enabling CCR The Steps We Took

Step 1 Install and Configuration New Elasticsearch Cluster

First install Elasticsearch on your cluster nodes. Start by installing the required Java runtime. Then install the Elasticsearch RPM package. After installation, reload the system daemon. Enable the Elasticsearch service so that it starts automatically when the system boots.

Finally reset the password, for the superuser. For any nodes generate an enrollment token from the node. Use that token to reconfigure and add the nodes to the Elasticsearch cluster.

Step 2 Enable the License

Both clusters need a license that supports CCR. We started a trial license on both sides using the license API and verified it was active. This is a step but easy to forget if you’re moving fast.

// Start trial license on both clusters
curl -XPOST http://es-ip:9200/_license/start_trial?acknowledge=true -u elastic:password

//Verify license status
curl -X GET "http://es-ip:9200/_license" -u elastic:password

Step 3 Configure TLS Certificates

Secure communication between clusters means you must exchange CA certificates. We found this step essential because CCR communication happens over the transport layer and that layer is encrypted by default in Elasticsearch versions.

On the Leader Cluster

Export the CA certificate from the transport keystore with OpenSSL. Transfer it to follower cluster, we have used S3 bucket for that. Make sure to do this so the follower cluster can retrieve the file.

   // Export the CA certificate from the transport keystore
   openssl pkcs12 -in /etc/elasticsearch/certs/transport.p12 -cacerts  
   -nokey -out cluster-a-ca.crt

   // Upload to shared S3 location
   aws s3 cp cluster-a-ca.crt s3://bucket-name/ --region ap-south-1

On the Follower Cluster

Download the leader clusters CA certificate fetch the truststore password, from the Elasticsearch keystore. Then import the certificate into the local transport keystore using the keytool utility.

   // Download leader cluster's CA certificate
   aws s3 cp s3://bucket-name/cluster-a-ca.crt --region ap-south-1 .

   // Get the truststore password
   /bin/elasticsearch-keystore show xpack.security.transport.ssl.truststore.secure_password

   // Import the certificate into the local transport keystore
   keytool -importcert -keystore /etc/elasticsearch/certs/transport.p12 
     -storetype PKCS12 -alias cluster-a-ca -file cluster-a-ca.crt 
     -noprompt -storepass <truststore_password>

Repeat the same steps so the leader cluster also trusts the follower clusters CA certificate. Also double-check that the reverse steps are correct.

Configure Node Roles

On the follower cluster add remote_cluster_client to the node roles in elasticsearch.yml. This role is needed for any node that needs to talk to a cluster. This role is needed for any node that needs to talk to a cluster. Restart Elasticsearch on each node one, at a time to make the change take effect without stopping the cluster.

   node.roles: [master, data, remote_cluster_client]

Step 4: Add Remote Cluster

On the follower cluster set up the connection to the leader cluster using the cluster settings API. Give the seed nodes of the leader cluster turn on transport compression to save bandwidth and set skip_unavailable to false if you want availability.

curl -X PUT "http://es-ip:9200/_cluster/settings" 
  -H 'Content-Type: application/json' -u elastic:<password> 
  -d '{
    "persistent": {
      "cluster": {
        "remote": {
          "leader-cluster": {
            "seeds": [
              "leader-node-1:9300",
              "leader-node-2:9300"
            ],
            "transport.compress": true,
            "skip_unavailable": false
          }
        }
      }
    }
  }'

Step 5: Start Replication

Begin Replication Create a follower index on the follower cluster that follows the leader index. This is done with the CCR follow API, where you enter the name of the cluster and the name of the leader index.

curl -XPUT "http://es-ip:9200/<follower_index>/_ccr/follow?pretty"  
-H 'Content-Type: application/json' -u elastic:<password> 
-d '{
  "remote_cluster": "leader-cluster",
  "leader_index": "<leader_index>"
}'

Once replication starts check the index status and replication details. The follower index should show up in the list of indexes. The CCR stats API will show how the replication is going including any delay, between the leader and the follower.

// Check indices
curl -X GET "http://es-ip:9200/_cat/indices?pretty" -u elastic:<password>

// Check CCR stats
curl -X GET "http://es-ip:9200/_ccr/stats?pretty" -u elastic:<password>

Step 6: Monitor Replication

Monitoring must be done. The metrics that matter are Replication lag, the gap between the leader and follower checkpoints. Also watch for reads and writes because they show connectivity or permission problems. Then check sync status to see if Replication is actively paused or in an error state.

// Check follower stats for a specific index
curl -XGET http://es-ip:9200/<follower_index>/_ccr/stats?pretty -u elastic:password

Check out these metrics. It’s so much easier to catch a problem early than to fix it after it’s been quietly failing for hours.

Disabling CCR on the Follower Cluster

To disable CCR, there are four step process which need to perform:

Step 1: Pause the Follower Index

This stops ongoing replication without deleting anything.

curl -XPOST http://es-ip:9200/<follower_index>/_ccr/pause_follow -u elastic:password

Step 2: Close the Follower Index

Close the follower index to prepare it for the operation also you cannot unfollow an open index.

curl -X POST "http://es-ip:9200/<follower_index>/_close" -u elastic:password

Step 3: Unfollow the Index

This converts the follower into a regular read-write index. This step is irreversible. Once you unfollow it there is no way to go back to following without setting it up from scratch.

curl -XPOST http://es-ip:9200/<follower_index>/_ccr/unfollow -u elastic:password

Step 4: Access the Index again

Reopen the index to make it available for reads and writes.

curl -XPOST "http://es-ip:9200/<follower_index>/_open" -u elastic:password

Then we went back to the Basic license, as we didn’t need the trial anymore. We also commented out remote_cluster_client from the node roles on all nodes and restarted them one by one. Leaving that role enabled when you’re not using CCR is just unnecessary overhead.

// Revert to Basic license
curl -XPOST http://es-ip:9200/_license/start_basic?acknowledge=true -u elastic:password

// verify data
curl -XGET "http://es-ip:9200/_cat/indices?v&s=index&pretty" -u elastic:password

Best Practices

  • Use dedicated network links between datacenters if you can It makes a difference
  • Configure your timeouts based on your actual network latency not the defaults
  • Size your follower cluster properly It’s not just a passive receiver Its doing real work
  • Use TLS Create CCR users with minimal privileges instead of using the elastic superuser
  • Rotate certificates regularly

Conclusion

Cross-Cluster Replication gives a way to move Elasticsearch data between clusters with very little downtime. Here’s what we learned from our setup:

  • Cross-Cluster Replication replicates indices automatically from the leader cluster to the follower cluster.
  • You must set up TLS certificates correctly to keep communication, between clusters
  • The remote_cluster_client role needs to be set on the follower cluster nodes.
  • Unfollow actions can’t be undone so think carefully before switching over.

Following this guide helps you build a CCR migration plan. It keeps your data consistent. Your business runs smoothly during the move.

Tag

ccr CCRSetup ClusterReplication DataMigration elasticsearch

Leave a Reply

Your email address will not be published. Required fields are marked *