[DEPRECATED] Compute Partition

Example: Basic configuration

You configure the compute partition transformation in the Debezium connector’s Kafka Connect configuration. The configuration specifies the following parameters:

  • The data collection column to use to calculate the destination partition.

  • The maximum number of partitions permitted for the data collection.

The SMT only processes events that originate in specified data collections. Events from other data collections are ignored.

To configure a Debezium connector to route events to a specific partition, configure the ComputePartition SMT in the Kafka Connect configuration for the Debezium connector.

For example, you might add the following configuration in your connector configuration.

  1. ...
  2. topic.creation.default.partitions=2
  3. topic.creation.default.replication.factor=1
  4. ...
  5. transforms=ComputePartition
  6. transforms.ComputePartition.type=io.debezium.transforms.partitions.ComputePartition
  7. transforms.ComputePartition.partition.data-collections.field.mappings=inventory.products:name,inventory.orders:purchaser
  8. transforms.ComputePartition.partition.data-collections.partition.num.mappings=inventory.products:2,inventory.orders:2
  9. ...

The configuration in the preceding example enables partition computation for the products and orders data collections. The configuration specifies that the SMT uses the name column to compute the partition for the products data collection. The number of partitions is set to 2. The number of partitions that you specify must match the number of partitions that are specified in the Kafka topic configuration.

Based on the configuration in the example, for the following Products table, the SMT routes change events for all records that have the field name hammer to the same partition. That is, the items with id values 104, 105, and 106 are routed to the same partition.

Table 1. Products table

id

name

description

weight

101

scooter

Small 2-wheel scooter

3.14

102

car battery

12V car battery

8.1

103

12-pack drill bits

12-pack of drill bits with sizes ranging from #40 to #3

0.8

104

hammer

12oz carpenter’s hammer

0.75

105

hammer

14oz carpenter’s hammer

0.875

106

hammer

16oz carpenter’s hammer

1.0

107

rocks

box of assorted rocks

5.3

108

jacket

water-resistant black windbreaker

0.1

109

spare tire

24-inch spare tire

22.2

Configuration options

The following table lists the configuration options that you can use with the compute partition SMT.

Table 2. Partition routing SMT (ComputePartition) configuration options

Property

Default

Description

A comma-separated list of colon-delimited pairs that specify the columns to use for a specific data collection, for example, inventory.products:name,inventory.orders:purchaser.

A comma-separated list of colon-delimited pairs that specify the number of partitions to use for a specific data collection, for example, inventory.products:2,inventory.orders:3.