Apache Kafka examples

The following examples will help you understand how to use the different Apache Kafka components for Knative.

Prerequisites

All examples require:

  • A Kubernetes cluster with
  • Knative Eventing v0.9+
  • Knative Serving v0.9+
  • An Apache Kafka cluster

Setting up Apache Kafka

If you want to run the Apache Kafka cluster on Kubernetes, the simplest option is to install it by using Strimzi.

  1. Create a namespace for your Apache Kafka installation, like kafka:

    1. kubectl create namespace kafka

  2. Install the Strimzi operator, like:

    1. curl -L "https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.16.2/strimzi-cluster-operator-0.16.2.yaml" \
    2.   | sed 's/namespace: .*/namespace: kafka/' \
    3.   | kubectl -n kafka apply -f -

  3. Describe the size of your Apache Kafka installation in kafka.yaml, like:

    1. apiVersion: kafka.strimzi.io/v1beta2
    2. kind: Kafka
    3. metadata:
    4.   name: my-cluster
    5. spec:
    6.   kafka:
    7.     version: 2.4.0
    8.     replicas: 1
    9.     listeners:
    10.       plain: {}
    11.       tls: {}
    12.     config:
    13.       offsets.topic.replication.factor: 1
    14.       transaction.state.log.replication.factor: 1
    15.       transaction.state.log.min.isr: 1
    16.       log.message.format.version: "2.4"
    17.     storage:
    18.       type: ephemeral
    19.   zookeeper:
    20.     replicas: 3
    21.     storage:
    22.       type: ephemeral
    23.   entityOperator:
    24.     topicOperator: {}
    25.     userOperator: {}

  4. Deploy the Apache Kafka cluster

    1. $ kubectl apply -n kafka -f kafka.yaml

This will install a small, non-production, cluster of Apache Kafka. To verify your installation, check if the pods for Strimzi are all up, in the kafka namespace:

  1. $ kubectl get pods -n kafka
  2. NAME                                          READY   STATUS    RESTARTS   AGE
  3. my-cluster-entity-operator-65995cf856-ld2zp   3/3     Running   0          102s
  4. my-cluster-kafka-0                            2/2     Running   0          2m8s
  5. my-cluster-zookeeper-0                        2/2     Running   0          2m39s
  6. my-cluster-zookeeper-1                        2/2     Running   0          2m49s
  7. my-cluster-zookeeper-2                        2/2     Running   0          2m59s
  8. strimzi-cluster-operator-77555d4b69-sbrt4     1/1     Running   0          3m14s

NOTE: For production ready installs check Strimzi.

Installation script

If you want to install the latest version of Strimzi, in just one step, we have a script for your convenience, which does exactly the same steps that are listed above:

  1. $ ./kafka_setup.sh

Examples of Apache Kafka and Knative

A number of different examples, showing the KafkaSource, KafkaChannel and KafkaBinding can be found here: