Customization of Kafka Connect automatic topic creation

Disabling automatic topic creation for the Kafka broker

By default, the Kafka broker configuration enables the broker to create topics at runtime if the topics do not already exist. Topics created by the broker cannot be configured with custom properties. If you use a Kafka version earlier than 2.6.0, and you want to create topics with specific configurations, you must to disable automatic topic creation at the broker, and then explicitly create the topics, either manually, or through a custom deployment process.

Procedure

• In the broker configuration, set the value of auto.create.topics.enable to false.

Set up Kafka Connect

Automatic topic creation in Kafka Connect is controlled by the topic.creation.enable property. The default value for the property is true, enabling automatic topic creation, as shown in the following example:

topic.creation.enable = true

The setting for the topic.creation.enable property applies to all workers in the Connect cluster.

Kafka Connect automatic topic creation requires you to define the configuration properties that Kafka Connect applies when creating topics. You specify topic configuration properties in the Debezium connector configuration by defining topic groups, and then specifying the properties to apply to each group. The connector configuration defines a default topic creation group, and, optionally, one or more custom topic creation groups. Custom topic creation groups use lists of topic name patterns to specify the topics to which the group’s settings apply.

For details about how Kafka Connect matches topics to topic creation groups, see Topic creation groups. For more information about how configuration properties are assigned to groups, see Topic creation group configuration properties.

By default, topics that Kafka Connect creates are named based on the pattern server.schema.table, for example, dbserver.myschema.inventory.

Configuration

For Kafka Connect to create topics automatically, it requires information from the source connector about the configuration properties to apply when creating topics. You define the properties that control topic creation in the configuration for each Debezium connector. As Kafka Connect creates topics for event records that a connector emits, the resulting topics obtain their configuration from the applicable group. The configuration applies to event records emitted by that connector only.

Topic creation groups

A set of topic properties is associated with a topic creation group. Minimally, you must define a default topic creation group and specify its configuration properties. Beyond that you can optionally define one or more custom topic creation groups and specify unique properties for each.

When you create custom topic creation groups, you define the member topics for each group based on topic name patterns. You can specify naming patterns that describe the topics to include or exclude from each group. The include and exclude properties contain comma-separated lists of regular expressions that define topic name patterns. For example, if you want a group to include all topics that start with the string dbserver1.inventory, set the value of its topic.creation.inventory.include property to dbserver1\\.inventory\\.*.

Topic creation group configuration properties

The default topic creation group and each custom group is associated with a unique set of configuration properties. You can configure a group to include any of the Kafka topic-level configuration properties. For example, you can specify the cleanup policy for old topic segments, retention time, or the topic compression type for a topic group. You must define at least a minimum set of properties to describe the configuration of the topics to be created.

If no custom groups are registered, or if the include patterns for any registered groups don’t match the names of any topics to be created, then Kafka Connect uses the configuration of the default group to create topics.

See Configuring Debezium topics in the Debezium installation guide on generic topic configuration considerations.

Default group configuration

Before you can use Kafka Connect automatic topic creation, you must create a default topic creation group and define a configuration for it. The configuration for the default topic creation group is applied to any topics with names that do not match the include list pattern of a custom topic creation group.

Procedure

• To define properties for the topic.creation.default group, add them to the connector configuration JSON, as shown in the following example:

  {
      ...
      "topic.creation.default.replication.factor": 3,  (1)
      "topic.creation.default.partitions": 10,  (2)
      "topic.creation.default.cleanup.policy": "compact",  (3)
      "topic.creation.default.compression.type": "lz4"  (4)
       ...
  }

  You can include any Kafka topic-level configuration property in the configuration for the default group.

Custom group configuration

You can define multiple custom topic groups, each with its own configuration.

Procedure

• To define a custom topic group, add a topic.creation.*<group_name>*.include property to the connector JSON, and list the properties for the custom group after the group name.

  The following example shows sample configurations for the inventory and applicationlogs custom topic creation groups:

  {
      ...
      (1)
      "topic.creation.inventory.include": "dbserver1\\.inventory\\.*",  (2)
      "topic.creation.inventory.partitions": 20,
      "topic.creation.inventory.cleanup.policy": "compact",
      "topic.creation.inventory.delete.retention.ms": 7776000000,
      (3)
      "topic.creation.applicationlogs.include": "dbserver1\\.logs\\.applog-.*",  (4)
      "topic.creation.applicationlogs.exclude": "dbserver1\\.logs\\.applog-old-.*",  (5)
      "topic.creation.applicationlogs.replication.factor": 1,
      "topic.creation.applicationlogs.partitions": 20,
      "topic.creation.applicationlogs.cleanup.policy": "delete",
      "topic.creation.applicationlogs.retention.ms": 7776000000,
      "topic.creation.applicationlogs.compression.type": "lz4",
       ...
  }

Registering custom groups

After you specify the configuration for any custom topic creation groups, register the groups.

Procedure

• Register custom groups by adding the topic.creation.groups property to the connector JSON, and specifying a comma-separated list of groups.

  The following example registers the custom topic creation groups inventory and applicationlogs:

  {
      ...
      "topic.creation.groups": "inventory,applicationlogs",
      ...
  }

Completed configuration

The following example shows a completed configuration that includes the configuration for a default topic group, along with the configurations for an inventory and an applicationlogs custom topic creation group:

Example: Configuration for a default topic creation group and two custom groups

{
    ...
    "topic.creation.default.replication.factor": 3,
    "topic.creation.default.partitions": 10,
    "topic.creation.default.cleanup.policy": "compact",
    "topic.creation.default.compression.type": "lz4"
    "topic.creation.groups": "inventory,applicationlogs",
    "topic.creation.inventory.include": "dbserver1\\.inventory\\.*",
    "topic.creation.inventory.partitions": 20,
    "topic.creation.inventory.cleanup.policy": "compact",
    "topic.creation.inventory.delete.retention.ms": 7776000000,
    "topic.creation.applicationlogs.include": "dbserver1\\.logs\\.applog-.*",
    "topic.creation.applicationlogs.exclude": "dbserver1\\.logs\\.applog-old-.*",
    "topic.creation.applicationlogs.replication.factor": 1,
    "topic.creation.applicationlogs.partitions": 20,
    "topic.creation.applicationlogs.cleanup.policy": "delete",
    "topic.creation.applicationlogs.retention.ms": 7776000000,
    "topic.creation.applicationlogs.compression.type": "lz4"
}

Additional resources

For more information on topic auto-creation you can have a look at these resources:

• Debezium Blog: Auto-creating Debezium Change Data Topics

• Kafka Improvement Proposal about adding topic auto-creation to Kafka Connect: KIP-158 Kafka Connect should allow source connectors to set topic-specific settings for new topics