Apache Kafka supervisor reference

This topic contains configuration reference information for the Apache Kafka supervisor for Apache Druid. The following table outlines the high-level configuration options:

KafkaSupervisorIOConfig

Task Autoscaler Properties

Lag Based AutoScaler Strategy Related Properties

Idle Supervisor Configuration

Note that Idle state transitioning is currently designated as experimental.

When the supervisor enters the idle state, no new tasks will be launched subsequent to the completion of the currently executing tasks. This strategy may lead to reduced costs for cluster operators while using topics that get sporadic data.

The following example demonstrates supervisor spec with lagBased autoScaler and idle config enabled:

{
    "type": "kafka",
    "spec": {
      "dataSchema": {
        ...
      },
      "ioConfig": {
         "topic": "metrics",
         "inputFormat": {
             "type": "json"
         },
          "consumerProperties": {
             "bootstrap.servers": "localhost:9092"
          },
         "autoScalerConfig": {
             "enableTaskAutoScaler": true,
              "taskCountMax": 6,
              "taskCountMin": 2,
             "minTriggerScaleActionFrequencyMillis": 600000,
             "autoScalerStrategy": "lagBased",
              "lagCollectionIntervalMillis": 30000,
              "lagCollectionRangeMillis": 600000,
              "scaleOutThreshold": 6000000,
              "triggerScaleOutFractionThreshold": 0.3,
             "scaleInThreshold": 1000000,
             "triggerScaleInFractionThreshold": 0.9,
              "scaleActionStartDelayMillis": 300000,
              "scaleActionPeriodMillis": 60000,
              "scaleInStep": 1,
             "scaleOutStep": 2
         },
         "taskCount":1,
         "replicas":1,
         "taskDuration":"PT1H",
         "idleConfig": {
           "enabled": true,
           "inactiveAfterMillis": 600000 
         }
      },
     "tuningConfig":{
        ...
     }
    }
}

More on consumerProperties

Consumer properties must contain a property bootstrap.servers with a list of Kafka brokers in the form: <BROKER_1>:<PORT_1>,<BROKER_2>:<PORT_2>,.... By default, isolation.level is set to read_committed. If you use older versions of Kafka servers without transactions support or don’t want Druid to consume only committed transactions, set isolation.level to read_uncommitted.

In some cases, you may need to fetch consumer properties at runtime. For example, when bootstrap.servers is not known upfront, or is not static. To enable SSL connections, you must provide passwords for keystore, truststore and key secretly. You can provide configurations at runtime with a dynamic config provider implementation like the environment variable config provider that comes with Druid. For more information, see DynamicConfigProvider.

For example, if you are using SASL and SSL with Kafka, set the following environment variables for the Druid user on the machines running the Overlord and the Peon services:

export KAFKA_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username='admin_user' password='admin_password';"
export SSL_KEY_PASSWORD=mysecretkeypassword
export SSL_KEYSTORE_PASSWORD=mysecretkeystorepassword
export SSL_TRUSTSTORE_PASSWORD=mysecrettruststorepassword

        "druid.dynamic.config.provider": {
          "type": "environment",
          "variables": {
            "sasl.jaas.config": "KAFKA_JAAS_CONFIG",
            "ssl.key.password": "SSL_KEY_PASSWORD",
            "ssl.keystore.password": "SSL_KEYSTORE_PASSWORD",
            "ssl.truststore.password": "SSL_TRUSTSTORE_PASSWORD"
          }
        }
      }

Verify that you’ve changed the values for all configurations to match your own environment. You can use the environment variable config provider syntax in the Consumer properties field on the Connect tab in the Load Data UI in the web console. When connecting to Kafka, Druid replaces the environment variables with their corresponding values.

Note: You can provide SSL connections with Password Provider interface to define the keystore, truststore, and key, but this feature is deprecated.

Specifying data format

Kafka indexing service supports both inputFormat and parser to specify the data format. Use the inputFormat to specify the data format for Kafka indexing service unless you need a format only supported by the legacy parser.

Supported inputFormats include:

• csv
• delimited
• json
• kafka
• avro_stream
• avro_ocf
• protobuf

For more information, see Data formats. You can also read thrift formats using parser.

KafkaSupervisorTuningConfig

The tuningConfig is optional and default parameters will be used if no tuningConfig is specified.

IndexSpec

Bitmap types

For Roaring bitmaps:

For Concise bitmaps:

SegmentWriteOutMediumFactory