1. Home
2. Cluster Linking

Looking for Confluent Platform Cluster Linking docs? This page describes Cluster Linking on Confluent Cloud. If you are looking for Confluent Platform documentation, check out Cluster Linking on Confluent Platform.

Cluster Linking Quick Start

In this quick how-to, you will create a multi-region or multi-cloud architecture with just a few commands.

Setup

Get the latest version of Confluent Cloud

To start off, you’ll need the latest version of Confluent Cloud.

• No Confluent Cloud yet? You can get it in two commands in your terminal window. (You can replace ~/.local/bin in both commands with a different directory, if you wish.)

  curl -L --http1.1 https://cnfl.io/ccloud-cli | sh -s -- -b ~/.local/bin
  

  export PATH=~/.local/bin:$PATH;
  

• Got Confluent Cloud? Make sure it’s up-to-date. If you already have Confluent Cloud installed, just use ccloud update to get the latest version of the Confluent Cloud CLI with new Cluster Linking commands and tools.

  ccloud update
  

Create source and destination clusters

Next, you’ll need two Confluent Cloud clusters. Data will flow from a “source” cluster to a “destination” cluster, which you’ll want to put in a different region or cloud.

The source cluster can be a Basic cluster, Standard cluster, or Dedicated cluster cluster with Internet networking (available over the public internet).

If you don’t already have a cluster you want to use, you can spin one up from the Confluent Cloud web UI or directly from the Confluent Cloud CLI with this command:

ccloud kafka cluster create ClusterLinkingSource --type basic --cloud aws --region us-west-2

Your destination cluster must be a Dedicated cluster with public internet networking.

If you don’t already have a dedicated cluster you want to use as the destination, you can create one from the Confluent Cloud web UI or directly from the Confluent Cloud CLI with this command:

ccloud kafka cluster create ClusterLinkingDestination --type dedicated --cloud aws --region us-east-1 --cku 1 --availability single-zone

Tip

• Source and destination clusters should be in different regions to effectively demo Cluster Linking capabilities, so choose a different region or cloud for your destination cluster than your source cluster for a multi-region or multi-cloud setup. (For example, you might have a source cluster in Northern California, and a destination cluster in Northern Virginia.)
• A Confluent Cloud cluster has an hourly charge, and charges for any data into, out of, or stored on the cluster. So, running this tutorial will accrue some charges. If you are only using the clusters for this demo, make sure to delete them once you have finished this walkthrough. This is covered in Teardown.

Replicate Data Across Regions

Now that you have clusters coast-to-coast, you can geo-replicate some data.

Save cluster IDs and source endpoint

You will use your cluster details in each of the next few commands (when you create a cluster link and test out mirror topics), so put these in a handy place. Save your source cluster ID, your source cluster Endpoint, and your destination cluster ID as local variables in your terminal window.

You can get the IDs of your cluster(s) with ccloud kafka cluster list, and the source cluster’s Endpoint with ccloud kafka cluster describe <source_id>. If you’ve just created the clusters, you will get full descriptions of them with all this information as output to the ccloud kafka cluster create commands.

Copy each of the following commands and substitute in your cluster IDs and endpoints to save them as local variables.

source_id=lkc-XXXXX

source_endpoint=SASL_SSL://pkc-XXXXX.us-west-2.aws.confluent.cloud:9092

destination_id=lkc-XXXXX

Create a cluster link

To move data from one region to another, you’ll first create a cluster link. A cluster link is a persistent, one-way bridge from one cluster to another.

The cluster link needs an API key to read data from your source cluster.

If you need an API key, create one like this:

ccloud api-key create --resource $source_id

Important

If you’re creating a cluster link in production, or if the two clusters belong to different people, then you should use a service account API key instead.

Using the API key and secret as <api-key> and <api-secret>, create a cluster link named “my-link” like this:

ccloud kafka link create my-link --cluster $destination_id \
    --source-cluster-id $source_id \
    --source-bootstrap-server $source_endpoint \
    --source-api-key <api-key> --source-api-secret <api-secret>

Note

There is an hourly charge to run a cluster link, so delete it when you are done with it. This is covered in Teardown.

Create source and mirror topics

Now, that you’ve got a link running, try it out!

In Confluent, data is stored in topics. To move data across clusters, start with a topic on the source cluster, then use your cluster link to create a copy of it (a “mirror topic”) on the destination cluster.

Mirror topics reflect all data from their source topics. Consumers can read from mirror topics, giving them a local copy of all events contained in the topic. Mirror topics sync their source topic configurations, and stay up to date; so you don’t need to set up or change any configs on your mirror topics.

Tip

Mirror topics are read-only, so don’t try to produce to one!

Specify the API key to use on the source cluster for producing and consuming data from the CLI:

ccloud api-key use <source-api-key> --resource $source_id

You’ll need a source topic. Make a one-partition topic called “topic-to-link” on your source cluster:

ccloud kafka topic create topic-to-link --cluster $source_id --partitions 1

Then, put some data into it

seq 1 10 | ccloud kafka topic produce topic-to-link --cluster $source_id

This produced the numbers 1 through 10 to your source cluster (which could be, for example, to AWS in Northern California).

You can mirror that data to your destination region (for example, GCP in Northern Virginia), in one command:

ccloud kafka mirror create topic-to-link --cluster $destination_id --link my-link

You just geo-replicated data!

Consume from the mirror topic

Now, make use of your geo-replicated data by consuming those on the destination cluster.

You may need to first create an API key and secret for the CLI to use with the destination cluster:

ccloud api-key create --resource <destination-cluster-id>

ccloud api-key use <destination-api-key> --resource <destination-cluster-id>

Now, read from the mirror topic.

ccloud kafka topic consume topic-to-link --cluster $destination_id --from-beginning

Here is a copy of the command and the output you will see:

ccloud kafka topic consume topic-to-link --cluster $destination_id --from-beginning
Starting Kafka Consumer. Use Ctrl-C to exit.
1
2
3
4
5
6
7
8
9
10

Tip

You can quit the consumer at any time by hitting Ctrl + C at the same time.

Congrats! You created a multi-region or multi-cloud real-time streaming architecture.

Go Exploring

Try producing more messages to topic-to-link on your source cluster and consuming them from its mirror on your destination cluster.

You can create more mirror topics; as many as you want, using the same cluster link.

Log on to the Confluent Cloud web UI, and view the clusters you created from there.

You should be able to view the messages on both the source and destination (mirror) topics. To navigate to topic messages, select a cluster, click Topics, and select the Messages tab.

Tip

Choose “Jump to offset” or “Jump to timestamp”, type 1 and select 1/Parition 0. This shows all the messages on partition 1.

Teardown

When you are ready to quit the demo, don’t forget to tear down the resources so as not to incur hourly charges.

1. Delete topic-to-link on both clusters with these commands:

   ccloud kafka topic delete topic-to-link --cluster $destination_id
   

   ccloud kafka topic delete topic-to-link --cluster $source_id
   

2. Delete any other mirror topics.

   If you created more mirror topics, you’ll need to delete those, too.

   You can see a list of all of the mirror topics on your destination cluster with:

   ccloud kafka mirror list --cluster $destination_id
   

   Then, use this command to delete each mirror topic.

   ccloud kafka topic delete <topic-name> --cluster $destination_id
   

3. Delete the cluster link(s).

   Once all of the mirror topics are gone, you can delete the cluster link on your destination cluster:

   ccloud kafka link delete my-link --cluster $destination_id
   

   If you created more cluster links, you can see all of the cluster links going to your destination cluster with this command:

   ccloud kafka link list --cluster $destination_id
   

   Delete any additional cluster links.

4. Delete any clusters you no longer need.

   If you were using existing Confluent Cloud clusters that you want to continue to use, then you’re done!

   If you spun new ones up for this demo, you can delete them with the command ccloud kafka cluster delete <cluster-id>.

   Note

   Be careful; once you delete a cluster, you can’t get it back.

   If you were following along with the demo and created new clusters, just use $destination_id and $source_id:

   ccloud kafka cluster delete $destination_id
   

   ccloud kafka cluster delete $source_id
   

What’s Next?

Try out these other Cluster Linking use cases and tutorials:

• Share Data Across Clusters, Regions, and Clouds
• Disaster Recovery and Failover

Learn more about these concepts and configurations:

• Mirror Topics
• Security for Cluster Linking on Confluent Cloud