kafka-logger

Description

The kafka-logger Plugin is used to push logs as JSON objects to Apache Kafka clusters. It works as a Kafka client driver for the ngx_lua Nginx module.

It might take some time to receive the log data. It will be automatically sent after the timer function in the batch processor expires.

Attributes

This Plugin supports using batch processors to aggregate and process entries (logs/data) in a batch. This avoids the need for frequently submitting the data. The batch processor submits data every 5 seconds or when the data in the queue reaches 1000. See Batch Processor for more information or setting your custom configuration.

IMPORTANT

The data is first written to a buffer. When the buffer exceeds the batch_max_size or buffer_duration attribute, the data is sent to the Kafka server and the buffer is flushed.

If the process is successful, it will return true and if it fails, returns nil with a string with the “buffer overflow” error.

meta_format example

• default:

  {
    "upstream": "127.0.0.1:1980",
    "start_time": 1619414294760,
    "client_ip": "127.0.0.1",
    "service_id": "",
    "route_id": "1",
    "request": {
      "querystring": {
        "ab": "cd"
      },
      "size": 90,
      "uri": "/hello?ab=cd",
      "url": "http://localhost:1984/hello?ab=cd",
      "headers": {
        "host": "localhost",
        "content-length": "6",
        "connection": "close"
      },
      "body": "abcdef",
      "method": "GET"
    },
    "response": {
      "headers": {
        "connection": "close",
        "content-type": "text/plain; charset=utf-8",
        "date": "Mon, 26 Apr 2021 05:18:14 GMT",
        "server": "APISIX/2.5",
        "transfer-encoding": "chunked"
      },
      "size": 190,
      "status": 200
    },
    "server": {
      "hostname": "localhost",
      "version": "2.5"
    },
    "latency": 0
  }

• origin:

  GET /hello?ab=cd HTTP/1.1
  host: localhost
  content-length: 6
  connection: close
  abcdef

Metadata

You can also set the format of the logs by configuring the Plugin metadata. The following configurations are available:

IMPORTANT

Configuring the Plugin metadata is global in scope. This means that it will take effect on all Routes and Services which use the kafka-logger Plugin.

The example below shows how you can configure through the Admin API:

note

You can fetch the admin_key from config.yaml and save to an environment variable with the following command:

admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')

curl http://127.0.0.1:9180/apisix/admin/plugin_metadata/kafka-logger -H "X-API-KEY: $admin_key" -X PUT -d '
{
    "log_format": {
        "host": "$host",
        "@timestamp": "$time_iso8601",
        "client_ip": "$remote_addr"
    }
}'

With this configuration, your logs would be formatted as shown below:

{"host":"localhost","@timestamp":"2020-09-23T19:05:05-04:00","client_ip":"127.0.0.1","route_id":"1"}
{"host":"localhost","@timestamp":"2020-09-23T19:05:05-04:00","client_ip":"127.0.0.1","route_id":"1"}

Enable Plugin

The example below shows how you can enable the kafka-logger Plugin on a specific Route:

curl http://127.0.0.1:9180/apisix/admin/routes/5 -H "X-API-KEY: $admin_key" -X PUT -d '
{
    "plugins": {
       "kafka-logger": {
           "brokers" : [
             {
               "host" :"127.0.0.1",
               "port" : 9092
             }
            ],
           "kafka_topic" : "test2",
           "key" : "key1",
           "batch_max_size": 1,
           "name": "kafka logger"
       }
    },
    "upstream": {
       "nodes": {
           "127.0.0.1:1980": 1
       },
       "type": "roundrobin"
    },
    "uri": "/hello"
}'

This Plugin also supports pushing to more than one broker at a time. You can specify multiple brokers in the Plugin configuration as shown below:

 "brokers" : [
    {
      "host" :"127.0.0.1",
      "port" : 9092
    },
    {
      "host" :"127.0.0.1",
      "port" : 9093
    }
],

Example usage

Now, if you make a request to APISIX, it will be logged in your Kafka server:

curl -i http://127.0.0.1:9080/hello

Delete Plugin

To remove the kafka-logger Plugin, you can delete the corresponding JSON configuration from the Plugin configuration. APISIX will automatically reload and you do not have to restart for this to take effect.

curl http://127.0.0.1:9180/apisix/admin/routes/1  -H "X-API-KEY: $admin_key" -X PUT -d '
{
    "methods": ["GET"],
    "uri": "/hello",
    "plugins": {},
    "upstream": {
        "type": "roundrobin",
        "nodes": {
            "127.0.0.1:1980": 1
        }
    }
}'