Running a Logging Pipeline Locally

You may wish to test a logging pipeline locally to observe how it deals with log messages. The following is a walk-through for running Fluent Bit and Elasticsearch locally with Docker Compose which can serve as an example for testing other plugins locally.

Create a Configuration File

Refer to the Configuration File section to create a configuration to test.

fluent-bit.conf:

  1. [INPUT]
  2.   Name dummy
  3.   Dummy {"top": {".dotted": "value"}}
  5. [OUTPUT]
  6.   Name es
  7.   Host elasticsearch
  8.   Replace_Dots On

Docker Compose

Use Docker Compose to run Fluent Bit (with the configuration file mounted) and Elasticsearch.

docker-compose.yaml:

  1. version: "3.7"
  3. services:
  4.   fluent-bit:
  5.     image: fluent/fluent-bit
  6.     volumes:
  7.       - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
  8.     depends_on:
  9.       - elasticsearch
  10.   elasticsearch:
  11.     image: elasticsearch:7.6.2
  12.     ports:
  13.       - "9200:9200"
  14.     environment:
  15.       - discovery.type=single-node

View indexed logs

To view indexed logs run:

  1. curl "localhost:9200/_search?pretty" \
  2.   -H 'Content-Type: application/json' \
  3.   -d'{ "query": { "match_all": {} }}'

To “start fresh”, delete the index by running:

  1. curl -X DELETE "localhost:9200/fluent-bit?pretty"