1. Home
2. Cluster Linking

Looking for Confluent Platform Cluster Linking docs? This page describes Cluster Linking on Confluent Cloud. If you are looking for Confluent Platform documentation, check out Cluster Linking on Confluent Platform.

Security for Cluster Linking on Confluent Cloud

Security Model

The Cluster Linking security model separates the permissions and credentials needed for each step of its lifecycle. This paradigm enables many teams to use cluster link operations on the topics that they already own. It also allows the owner of one cluster to share data over a cluster link with another team or organization, while only giving that team or organization READ access to the shareable, mirror topics.

Advantages

The advantages of this model are:

• The access control lists (ACLs) needed to create a cluster link and mirror topics can be managed by the owner of the destination cluster. If you use role-based access control (RBAC), you can simply enable ClusterAdmin, EnvAdmin, or OrgAdmin roles to give users or applications permissions on the destination cluster. These roles are described in Confluent Cloud Cluster Authorization using Role-based Access Control (RBAC). Operations performed on a cluster link and a mirror topic are available in Confluent Cloud Audit Logs. Alternatively, you can simply specify the ACLs.
• The cluster link has its own credentials and ACLs on the source cluster. You can manage its READ access to the source cluster just like any other Apache Kafka® consumer. So, you can use your existing security workflows with cluster links.
  • If the source cluster is a Confluent Cloud cluster, it’s best to create a service account for the cluster link. You will issue source cluster credentials (API key and secret) to the cluster link’s service account.
  • If your source cluster is a Confluent Platform or open-source Apache Kafka cluster, the cluster link can use SASL/PLAIN or SASL/SCRAM. It cannot use Kerberos or OAuth.
• The cluster link itself requires only read ACLs on the source cluster. No ACLs or credentials are required on the destination cluster in Confluent Cloud. This simplifies management.
• When sharing data between teams, the owner of the source cluster can limit READ access to the mirror topics on the destination cluster on a topic-by-topic basis.

When sending data between the source and destination clusters, Cluster Linking uses Transport Layer Security (TLS) to encrypt data in transit.

If configured as such, Cluster Linking can also sync all client ACLs, or a configurable subset, from the source cluster to the destination cluster.

Building Blocks

The building blocks of Cluster Linking are:

• A cluster link, which connects a source cluster to a destination cluster
• A mirror topic, which exists on the destination cluster, and receives a copy of all messages that go to its source topic
• A source topic, which is a regular topic that exists on the source cluster and is the source of data for the mirror topic

Cluster Linking Lifecycle and Permissions

Each phase of the Cluster Linking lifecycle requires different permissions. The phases are:

• A user or client application creates a cluster link.
• A user or client application creates a mirror topic using the cluster link.
• The cluster link reads topic data, consumer offsets, and ACLs from the source cluster.
• The cluster link writes topic data, consumer offsets, and ACLs to the destination cluster.
• A user or client application modifies, promotes, or deletes the mirror topic.
• A user or client application modifies or deletes the cluster link.

Creating or modifying a cluster link

To create or modify a cluster link, a user must be authenticated and authorized by either a service account or a Confluent Cloud user account for role-based permissions on specified resources.

If the user is authenticated with a Confluent Cloud user account, they must have the ClusterAdmin, EnvAdmin, or OrgAdmin RBAC role over the destination cluster in order to be authorized to create, list, view, or modify cluster links.

If the user or client application is authenticated with a service account, then their service account needs an ACL to allow them to ALTER the destination cluster. To list the cluster links that exist on a destination cluster, their service account needs an ACL to allow them to DESCRIBE the destination cluster.

When you create a cluster link, you must give it a way to authenticate with the source cluster. The permissions for that are described below.

Creating or modifying a mirror topic

To create or modify a cluster link, a user must be authenticated and authorized by either a service account or a Confluent Cloud user account.

If the user is authenticated with a Confluent Cloud user account, they must have the ClusterAdmin, EnvAdmin, or OrgAdmin RBAC role over the destination cluster in order to be authorized to create or modify mirror topics.

If the user or client application is authenticated with a service account, then their service account needs two ACLs on the destination cluster:

• An ACL to allow them to ALTER the destination cluster.
• An ACL to allow them to CREATE the mirror topic. This ACL can be one that allows them to create all topics, or it can be scoped by topic name(s).

ACLs required to list and describe mirror topics

The list mirrors command lists the mirror topics on a cluster and/or a cluster link. You can call this command through the Confluent Cloud API for Cluster Linking with `REST`: `GET /kafka/v3/clusters/<cluster-id>/links/<link-name>/mirrors` or from the Confluent Cloud CLI with ccloud kafka mirror list.

The describe mirror command gives information about a specific mirror topic. You can call this command through the Confluent Cloud API with `REST`: `GET /kafka/v3/clusters/<cluster-id>/links/<link-name>/mirrors/<mirror-topic>` or from the Confluent Cloud CLI with ccloud kafka mirror describe <mirror-topic>.

To list and describe mirror topics, you need one of the following ACLs:

• The DESCRIBE ACL on a Cluster resource, which provides access to both list and describe for all topics on the cluster.
• The DESCRIBE ACL on specific Topic(s), which provides access to both list and describe for the specified topics only.

Permissions for the cluster link to read from the source cluster

For every mirror topic on a cluster link, the cluster link reads topic data from the respective source topic. If configured to sync consumer offsets or ACLs, the cluster link also reads those from the source cluster. To read this, the cluster link must authenticate with the source cluster and be authorized to perform these actions.

Cluster link authentication and authorization for Confluent Cloud source clusters

If the source cluster is a Confluent Cloud cluster, that means the cluster link must have:

• A service account that represents it, and
• An API key and secret to use with the source cluster

As a security best practice, you should create a service account to represent each cluster link. For example:

ccloud service-account create <name> --description "For my first cluster link"

The service account will be tied to the ACLs on the source cluster so that the cluster link can read the appropriate data.

You will then create an API key and secret on the source cluster for the service account, as shown in this example:

ccloud api-key create --resource <source-cluster-id> --service-account <service-account-id>

When you create the cluster link, you will give it the API key and secret, which it will use to authenticate with the source cluster.

It is possible to give the cluster link an API key from a shared service account or a user’s Confluent Cloud account. This is not recommended for source clusters with sensitive data, or for production use.

In order to be authorized to read data and metadata, the cluster link’s service account must have these ACLs on the source cluster:

• Allowed to READ and DESCRIBE_CONFIGS for all topics you want to mirror (“source topics”). This will let the cluster link mirror topic data from the source topic to the mirror topic. You could allow the link to read all topics by passing in *, or for specific topics whose names match a prefix, or for specific topic names. Here is an example CLI command to give the cluster link READ access to all topics:

  ccloud kafka acl create --allow --service-account <service-account-id> --operation READ --operation DESCRIBE_CONFIGS --topic "*" --cluster <source-cluster-id>
  

• To sync ACLs (optional), the cluster link must have permissions to DESCRIBE the source cluster. Here is an example of how to specify these permissions.

  ccloud kafka acl create --allow --service-account <service-account-id> --operation DESCRIBE --cluster-scope --cluster <source-cluster-id>
  

• To sync consumer group offsets (optional), the cluster link must have permissions to DESCRIBE source topics, and READ and DESCRIBE consumer groups on the source cluster. Here is an example of how to specify these permissions, each of which has to be specified in a separate command:

  ccloud kafka acl create --allow --service-account <service-account-id> --operation DESCRIBE --topic "*" --cluster <source-cluster-id>
  

  ccloud kafka acl create --allow --service-account <service-account-id> --operation READ --operation DESCRIBE --consumer-group "*" --cluster <source-cluster-id>
  

Summary of ACLs a cluster link needs on the source cluster

Capability	Required ACLs on Source Cluster
Mirror data from a source topic to a mirror topic	READ and DESCRIBE_CONFIGS for the source topic
Sync ACLs from source cluster to destination cluster	DESCRIBE on the source cluster
Sync a consumer group’s offsets from a source topic to a mirror topic	DESCRIBE on the source topic, READ and DESCRIBE the consumer group on the source cluster

Providing the cluster link with an API key and secret

For testing or development, you can use a short-hand way to create a cluster link with an API key and secret in the Confluent Cloud CLI:

ccloud kafka link create <link_name> --source-api-key <api-key> --source-api-secret <api-secret>

Using the CLI short-hand is not recommended for a cluster link in production because the terminal history could reveal your API key and secret.

Rather, you should pass the following three properties to your cluster link’s configuration:

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='<api-key>' password='<api-secret>';
security.protocol=SASL_SSL
sasl.mechanism=PLAIN

If you create the cluster link with the CLI, you can put those three lines in a config file (along with any other properties you want to configure on the link) and pass the file name to the command with the --config-file flag. For example:

ccloud kafka link create <link_name> --config-file <path/to/file.config> ...

If you create the command via a REST API call, you can add the three properties to your array of configs:

{
  "source_cluster_id": "<source cluster's ID>",
  "configs": [
    {
      "name": "security.protocol",
      "value": "SASL_SSL"
    },
    {
      "name": "sasl.mechanism",
      "value": "PLAIN"
    },
    {
      "name": "sasl.jaas.config",
      "value": "org.apache.kafka.common.security.plain.PlainLoginModule required username='<api-key>' password='<api-secret>';"
    },
    ...
  ],
  ...
}

Note

The security credentials stored on the link are encrypted and cannot be retrieved by a user. There is no way for someone to query the cluster link to get its API key or secret.

Changing or rotating the security credentials on a cluster link

If you need to change (or “rotate”) the security credentials on a cluster link, you can simply update the link’s sasl.jaas.config property with the new API key and secret. For example, if you create a new file called new-creds.config with an updated API key and secret:

# new-creds.config has this:
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='<api-key>' password='<api-secret>';

Then, you can run this command to specify that the cluster link use the new credentials in new-creds.config

ccloud kafka link update <link-name> --config-file new-creds.config

Caution

If the cluster link loses authorization to read data on the source cluster, mirrored topics will permanently fail.

If your cluster link’s ACLs to read source data are revoked, its mirror topics will go into a permanent FAILED state. The mirror topics will no longer mirror data from the source cluster, even if the cluster link’s authorization is later restored. If you want to restore mirroring to FAILED mirror topics, you will need to delete and recreate them.

Writing to the destination cluster

On Confluent Cloud, cluster links don’t need any extra permissions to write data to the destination cluster. Because they exist on the destination cluster, they already have the appropriate permissions.

Transport Layer Security

For a cluster link between two Confluent Cloud clusters, data in transit is encrypted through one-way TLS V1.2 (in the Java community, this is referred to as one-way SSL).

For a cluster link from a Confluent Platform or Apache Kafka cluster to a Confluent Cloud cluster, data-in-transit must use one-way TLS (in the Java community, this is referred to as one-way SSL). It will default to the latest version of TLS that both clusters have enabled. TLS 1.2 and 1.3 are enabled by default in Apache Kafka.

Older versions of SSL can be enabled, but this is discouraged due to known security vulnerabilities.

Practical Examples

To see practical examples of how you might model security for different Cluster Linking deployments, refer to these use case tutorials:

• Data Sharing
• Disaster Recovery and Failover

Audit Logs

Whenever a user or application tries to create or delete a cluster link or a mirror topic, authentication and authorization events are created in Confluent Cloud Audit Logs. This allows you to monitor Cluster Linking access and activity throughout Confluent Cloud.

Suggested Reading

• Manage Accounts and Access
• Service Accounts for Confluent Cloud
• ACL concepts