1. Home
  2. Log and Monitor in Confluent Cloud
  3. Audit Logs for Confluent Cloud

Confluent Cloud Audit Log Concepts

Audit logs provide a way to capture, protect, and preserve Kafka authentication and authorization activity into topics in Kafka clusters on Confluent Cloud. Specifically, audit logs record the runtime decisions of the permission checks that occur as users and service accounts connect to clusters and attempt to take actions that are protected by ACLs or RBAC.

Each auditable event includes information about who tried to do what, when they tried, and whether or not the system gave permission to proceed.

The primary value of audit logs is that they provide data you can use to assess security risks in your Confluent Cloud clusters. They contain all of the information necessary to follow a user’s interaction with your Confluent Cloud clusters, and provide a way to:

  • Track user and application access
  • Identify abnormal behavior and anomalies
  • Proactively monitor and resolve security risks

You can use Splunk, S3, and other sink connectors to move your audit log data to a target platform for analysis. Note that you cannot consume audit logs directly from a managed connector. Rather, you must either use a self-managed connector to consume audit logs, or mirror the audit logs into a managed Kafka cluster using Replicator or Cluster Linking on Confluent Cloud.

Within Confluent Cloud, all audit log messages from your clusters are retained for seven days on an independent cluster. Users cannot modify, delete, nor produce messages directly to the audit log topic, and to consume the messages, users must have an API key specific to the audit log cluster.

Audit logs are emitted at the time of event occurrence; however, short delays or gaps during operational maintenance are possible, although rare.

Auditable events

Confluent Cloud audit logs include two kinds of events: authentication events that are sent when a client connects to a Kafka cluster, and authorization events that are sent when the Kafka cluster checks to verify whether or not a client is allowed to take an action.

Note

Users may attempt to authorize a task solely to find out if they can perform the task, but not follow through with it. In these instances, the authorization is still captured in the audit log.

Confluent Cloud audit logs capture the following events:

Event name Description
mds.Authorize An RBAC authorization is being checked.
kafka.AlterConfigs A Kafka configuration is being altered/updated.
kafka.AlterMirrors The properties of a mirror topic that exists on a Cluster Link to this cluster are being altered.
kafka.Authentication A client has connected to the Kafka cluster using an API key or token.
kafka.CreateAcls A Kafka broker ACL is being created.
kafka.CreateClusterLinks A cluster link is being created between this cluster and another cluster.
kafka.CreatePartitions Partitions are being added to a topic.
kafka.CreateTopics A topic is being created.
kafka.DeleteAcls A Kafka broker ACL is being deleted.
kafka.DeleteClusterLinks A cluster link is being deleted.
kafka.DeleteGroups A Kafka group is being deleted.
kafka.DeleteRecords A Kafka record is being deleted. Commonly seen on ksqlDB internal topics for repartitioning.
kafka.DeleteTopics A Kafka topic is being deleted.
kafka.IncrementalAlterConfigs A dynamic configuration of a Kafka broker is being altered.
kafka.OffsetDelete A committed offset for a partition in a consumer group is being deleted.

All Confluent Cloud audit log messages are captured in the audit log topic, confluent-audit-log-events.

The following example shows an authentication event that was sent when service account 306343 used the API key MAIDSRFG53RXYTKR to connect to the Kafka cluster lkc-6k8r8q:

{
    "id": "29ca0e51-fdcd-44bd-a393-43193432b614",
    "source": "crn://confluent.cloud/kafka=lkc-6k8r8q",
    "specversion": "1.0",
    "type": "io.confluent.kafka.server/authentication",
    "datacontenttype": "application/json",
    "subject": "crn://confluent.cloud/kafka=lkc-6k8r8q",
    "time": "2020-12-28T22:41:43.395Z",
    "data": {
        "serviceName": "crn://confluent.cloud/kafka=lkc-6k8r8q",
        "methodName": "kafka.Authentication",
        "resourceName": "crn://confluent.cloud/kafka=lkc-6k8r8q",
        "authenticationInfo": {
            "principal": "User:306343",
            "metadata": {
                "mechanism": "SASL_SSL/PLAIN",
                "identifier": "MAIDSRFG53RXYTKR"
            }
        },
        "result": {
            "status": "SUCCESS",
            "message": ""
        }
    }
}

See also: Audit Log Event Schema.

Audit log content

The following example shows the contents of an audit log message that is returned from a cluster when a user creates a new topic:

{
    "id": "fc0f727d-899a-4a22-ad8b-a866871a9d37",
    "source": "crn://confluent.cloud/kafka=lkc-a1b2c",
    "specversion": "1.0",
    "type": "io.confluent.kafka.server/authorization",
    "datacontenttype": "application/json",
    "subject": "crn://confluent.cloud/kafka=lkc-a2b2c",
    "time": "2021-01-01T12:34:56.789Z",
    "data": {
        "serviceName": "crn://confluent.cloud/kafka=lkc-a1b2c",
        "methodName": "kafka.CreateTopics",
        "resourceName": "crn://confluent.cloud/kafka=lkc-a1b2c/topic=departures",
        "authenticationInfo": {
            "principal": "User:123456"
        },
        "authorizationInfo": {
            "granted": true,
            "operation": "DescribeConfigs",
            "resourceType": "Topic",
            "resourceName": "departures",
            "patternType": "LITERAL",
            "superUserAuthorization": true
        },
        "request": {
            "correlationId": "123",
            "clientId": "adminclient-42"
        }
    }
}

Note

For more examples, see Audit Log Schema Examples.

The first section of the audit log displays information about the log message itself. Think of it as the envelope in which the message is delivered; it contains the following audit log implementation details:

Audit log entry Description
id A randomly-generated UUID that ensures uniqueness across all sources.
source Identifies the source location where the decision about whether or not the authorization was allowed. For events that occur on Kafka clusters, this will be a Confluent Resource Name (CRN) that identifies the cluster. For cluster management events, this will be a top-level CRN for Confluent Cloud.
specversion The version of CloudEvents in use (CloudEvents is a vendor-neutral specification that defines the format of event data).
type

The type of event that occurred. The value in this property corresponds with the presence or absence of certain other event properties. The two types available are:

  • io.confluent.kafka.server/authentication
  • io.confluent.kafka.server/authorization

For additional details, read the audit log event schema.
datacontenttype Identifies the CloudEvent format that the audit data is presented in (JSON).
subject Identifies the resource on which access has been allowed or denied.
time The time at which the event occurred.

The next section of the audit log displays data (data) you can use to assess security risks.

Audit log entry Description
serviceName The system that determined whether or not the authorization was allowed. This will be either the CRN of a Kafka cluster or a general CRN like crn://confluent.cloud/.
methodName The type of request that was made. For example, the methodName above shows that the user attempted to create a Kafka topic. This entry will include one of the event names listed above, and display either mds (if it’s an authorize event) or kafka (if any other event) before it. For example: mds.Authorize, and kafka.CreateTopics.
resourceName

The Confluent resource name (CRN) of the target of the request, such as a Kafka cluster, topic, or consumer group. In the example above, the resourceName indicates that the user attempted to create a topic in a uniquely-identified Kafka cluster.

For additional details, refer to CRN.
authenticationInfo Identifies the user or service account attempting the request.
authorizationInfo

Includes details about the decision that was made to authorize the request. The example above shows that the authorization request to create a topic in a Kafka cluster was granted.

If RBAC is used, the resource and role (if any) are included.

If ACLs are used, the ACL information is also included.
request

Includes details about request-related identifiers, including the following:

  • correlation_id: A user-supplied integer that is passed back in the response by the server, unmodified. The correlation_id is useful for matching a request and response between the client and server.
  • client_id: The ID of the client that committed the auditable action. For example, if an admin client with ID admin_client_1 creates a topic, then the client_id will be admin_client_1.

Audit Log Content Examples

The following example shows what an audit log message looks like when RBAC is configured:

{
  "id": "f07bdde7-c633-41c9-abab-5ff3539e9967",
  "source": "crn://confluent.cloud/",
  "specversion": "1.0",
  "type": "io.confluent.kafka.server/authorization",
  "datacontenttype": "application/json",
  "subject": "crn://confluent.cloud/organization=1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d/environment=env-1ab2c",
  "time": "2021-06-07T18:49:40.331Z",
  "data": {
    "serviceName": "crn://confluent.cloud/",
    "methodName": "mds.Authorize",
    "resourceName": "crn://confluent.cloud/organization=1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d/environment=env-1ab2c",
    "authenticationInfo": {
      "principal": "User:u-1abc2d"
    },
    "authorizationInfo": {
      "granted": true,
      "operation": "CreateCloudCluster",
      "resourceType": "Environment",
      "resourceName": "environment",
      "patternType": "LITERAL",
      "rbacAuthorization": {
        "role": "OrganizationAdmin",
        "scope": {
          "outerScope": [
            "organization=1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d"
          ],
          "clusters": {}
        }
      }
  }
}

The following example shows what an audit log message looks like when you are allowed to alter topic configurations based on an ACL:

{
    "id": "fc0f727d-899a-4a22-ad8b-a866871a9d37",
    "source": "crn://confluent.cloud/kafka=lkc-a1b2c",
    "specversion": "1.0",
    "type": "io.confluent.kafka.server/authorization",
    "datacontenttype": "application/json",
    "subject": "crn://confluent.cloud/kafka=lkc-a2b2c",
    "time": "2021-01-01T12:34:56.789Z",
    "data": {
        "serviceName": "crn://confluent.cloud/kafka=lkc-a1b2c",
        "methodName": "kafka.IncrementalAlterConfigs",
        "resourceName": "crn://confluent.cloud/kafka=lkc-a1b2c/topic=departures",
        "authenticationInfo": {
            "principal": "User:123456"
        },
        "authorizationInfo": {
            "granted": true,
            "operation": "AlterConfigs",
            "resourceType": "Topic",
            "resourceName": "departures",
            "patternType": "LITERAL",
            "superUserAuthorization": false,
            "aclAuthorization": {
                "permissionType": "ALLOW",
                "host": "*"
            }
        },
        "request": {
            "correlationId": "123",
            "clientId": "adminclient-42"
        }
    }
}

Confluent Resource Names (CRNs)

Confluent Resource Names (CRNs) provide a uniform way to uniquely identify resources such as a Kafka cluster, topic, or consumer group, and are used in audit logs.

Each CRN is a uniform resource identifier (URI) that uniquely identifies the associated Confluent resource and is prefixed with crn://.

In the CRN:

crn://confluent.cloud/organization=fc5ba16d-661d-474c-85df-c2a1ed26032c/environment=env-pqr45/cloud-cluster=lkc-a1b2c/kafka=lkc-a1b2c

Interpret the content as follows:

  • An authorization event occurred in Kafka cluster kafka=lkc-a1b2c
  • The Kafka cluster is in the environment environment=env-pqr45
  • The environment is in the organization organization=fc5ba16d-661d-474c-85df-c2a1ed26032c

The following CRNs identify some of the most commonly-used Confluent Cloud resources.

Cluster Resource CRN

This CRN identifies the Kafka cluster.

Example:

crn://confluent.cloud/organization=fc5ba16d-661d-474c-85df-c2a1ed26032c/environment=env-pqr45/cloud-cluster=lkc-a1b2c/kafka=lkc-a1b2c
crn://confluent.cloud/kafka=lkc-a1b2c
Topic CRN

This CRN identifies Confluent Cloud topics.

Example:

crn://confluent.cloud/kafka=lkc-a1b2c/topic=my-topic
Service Account CRN

This CRN identifies Confluent Cloud service accounts.

Examples:

crn://confluent.cloud/organization=fc5ba16d-661d-474c-85df-c2a1ed26032c/service-account=123456
crn://confluent.cloud/organization=fc5ba16d-661d-474c-85df-c2a1ed26032c/service-account=sa-x8y9z0
crn://confluent.cloud/organization=fc5ba16d-661d-474c-85df-c2a1ed26032c/service-account=%2A
Schema Registry CRN

This CRN identifies a Confluent Cloud Schema Registry.

Example:

crn://confluent.cloud/organization=fc5ba16d-661d-474c-85df-c2a1ed26032c/environment=env-pqr45/schema-registry=lsrc-m1n2o

For additional details about CRNs, refer to Confluent Resource Names.