Fluent Bit and Golang Plugins

Fluent Bit currently supports integration of Golang plugins built as shared objects for output plugins only. The interface for the Golang plugins is currently under development but is functional.

Getting Started

Compile Fluent Bit with Golang support, e.g:

  1. $ cd build/
  2. $ cmake -DFLB_DEBUG=On -DFLB_PROXY_GO=On ../
  3. $ make

Once compiled, we can see a new option in the binary -e which stands for external plugin, e.g:

  1. $ bin/fluent-bit -h
  2. Usage: fluent-bit [OPTION]
  4. Available Options
  5.   -c  --config=FILE    specify an optional configuration file
  6.   -d, --daemon        run Fluent Bit in background mode
  7.   -f, --flush=SECONDS    flush timeout in seconds (default: 5)
  8.   -i, --input=INPUT    set an input
  9.   -m, --match=MATCH    set plugin match, same as '-p match=abc'
  10.   -o, --output=OUTPUT    set an output
  11.   -p, --prop="A=B"    set plugin configuration property
  12.   -e, --plugin=FILE    load an external plugin (shared lib)
  13.   ...

Build a Go Plugin

The fluent-bit-go package is available to assist developers in creating Go plugins.

https://github.com/fluent/fluent-bit-go

At a minimum, a Go plugin looks like this:

  1. package main
  3. import "github.com/fluent/fluent-bit-go/output"
  5. //export FLBPluginRegister
  6. func FLBPluginRegister(def unsafe.Pointer) int {
  7.     // Gets called only once when the plugin.so is loaded
  8.     return output.FLBPluginRegister(ctx, "gstdout", "Stdout GO!")
  9. }
  11. //export FLBPluginInit
  12. func FLBPluginInit(plugin unsafe.Pointer) int {
  13.     // Gets called only once for each instance you have configured.
  14.     return output.FLB_OK
  15. }
  17. //export FLBPluginFlushCtx
  18. func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.char) int {
  19.     // Gets called with a batch of records to be written to an instance.
  20.     return output.FLB_OK
  21. }
  23. //export FLBPluginExit
  24. func FLBPluginExit() int {
  25.     return output.FLB_OK
  26. }
  28. func main() {
  29. }

the code above is a template to write an output plugin, it’s really important to keep the package name as main and add an explicit main() function. This is a requirement as the code will be build as a shared library.

To build the code above, use the following line:

  1. $ go build -buildmode=c-shared -o out_gstdout.so out_gstdout.go

Once built, a shared library called out\_gstdout.so will be available. It’s really important to double check the final .so file is what we expect. Doing a ldd over the library we should see something similar to this:

  1. $ ldd out_gstdout.so
  2.     linux-vdso.so.1 =>  (0x00007fff561dd000)
  3.     libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc4aeef0000)
  4.     libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc4aeb27000)
  5.     /lib64/ld-linux-x86-64.so.2 (0x000055751a4fd000)

Run Fluent Bit with the new plugin

  1. $ bin/fluent-bit -e /path/to/out_gstdout.so -i cpu -o gstdout