Developer guide for beginners on contributing to Fluent Bit

Assuming you have some basic knowledge of C, this guide should help you understand how to make code changes to Fluent Bit.

Libraries

Most external libraries are embedded in the project in the /lib folder. To keep its footprint low and make cross-platform builds simple, Fluent Bit attempts keep its dependency graph small.

The external library you are mostly likely to interact with is msgpack.

For crypto, Fluent Bit uses mbedtls.

Memory Management

When you write Fluent Bit code, you will use Fluent Bit’s versions of the standard C functions for working with memory:

Note that many types have a specialized create and destroy function. For example, flb_sds_create() and flb_sds_destroy() (more about this in the next section).

Strings

Fluent Bit has a stripped down version of the popular SDS string library. See flb_sds.h for the API.

In general, you should use SDS strings in any string processing code. SDS strings are fully compatible with any C function that accepts a null-terminated sequence of characters; to understand how they work, see the explanation on Github.

HTTP Client

Fluent Bit has its own network connection library. The key types and functions are defined in the following header files:

The following code demonstrates making an HTTP request in Fluent Bit:

  1. #include <fluent-bit/flb_upstream.h>
  2. #include <fluent-bit/flb_io.h>
  3. #include <fluent-bit/flb_http_client.h>
  4. #include <fluent-bit/flb_info.h>
  5. #include <fluent-bit/flb_config.h>
  7. #define HOST  "127.0.0.1"
  8. #define PORT  80
  10. static flb_sds_t make_request(struct flb_config *config)
  11. {
  12.     struct flb_upstream *upstream;
  13.     struct flb_http_client *client;
  14.     size_t b_sent;
  15.     int ret;
  16.     struct flb_upstream_conn *u_conn;
  17.     flb_sds_t resp;
  19.     /* Create an 'upstream' context */
  20.     upstream = flb_upstream_create(config, HOST, PORT, FLB_IO_TCP, NULL);
  21.     if (!upstream) {
  22.         flb_error("[example] connection initialization error");
  23.         return -1;
  24.     }
  26.     /* Retrieve a TCP connection from the 'upstream' context */
  27.     u_conn = flb_upstream_conn_get(upstream);
  28.     if (!u_conn) {
  29.         flb_error("[example] connection initialization error");
  30.         flb_upstream_destroy(upstream);
  31.         return -1;
  32.     }
  34.     /* Create HTTP Client request/context */
  35.     client = flb_http_client(u_conn,
  36.                              FLB_HTTP_GET, metadata_path,
  37.                              NULL, 0,
  38.                              FLB_FILTER_AWS_IMDS_V2_HOST, 80,
  39.                              NULL, 0);
  41.     if (!client) {
  42.         flb_error("[example] count not create http client");
  43.         flb_upstream_conn_release(u_conn);
  44.         flb_upstream_destroy(upstream);
  45.         return -1;
  46.     }
  48.     /* Perform the HTTP request */
  49.     ret = flb_http_do(client, &b_sent)
  51.     /* Validate return status and HTTP status if set */
  52.     if (ret != 0 || client->resp.status != 200) {
  53.         if (client->resp.payload_size > 0) {
  54.             flb_debug("[example] Request failed and returned: \n%s",
  55.                       client->resp.payload);
  56.         }
  57.         flb_http_client_destroy(client);
  58.         flb_upstream_conn_release(u_conn);
  59.         flb_upstream_destroy(upstream);
  60.         return -1;
  61.     }
  63.     /* Copy payload response to an output SDS buffer */
  64.     data = flb_sds_create_len(client->resp.payload,
  65.                               client->resp.payload_size);
  67.     flb_http_client_destroy(client);
  68.     flb_upstream_conn_release(u_conn);
  69.     flb_upstream_destroy(upstream);
  71.     return resp;
  72. }

An flb_upstream structure represents a host/endpoint that you want to call. Normally, you’d store this structure somewhere so that it can be re-used. An flb_upstream_conn represents a connection to that host for a single HTTP request. The connection structure should not be used for more than one request.

Linked Lists

Fluent Bit contains a library for constructing linked lists- mk_list. The type stores data as a circular linked list.

The mk_list.h header file contains several macros and functions for use with the lists. The example below shows how to create a list, iterate through it, and delete an element.

  1. #include <monkey/mk_core/mk_list.h>
  2. #include <fluent-bit/flb_info.h>
  4. struct item {
  5.     char some_data;
  7.     struct mk_list _head;
  8. };
  10. static int example()
  11. {
  12.     struct mk_list *tmp;
  13.     struct mk_list *head;
  14.     struct mk_list items;
  15.     int i;
  16.     int len;
  17.     char characters[] = "abcdefghijk";
  18.     struct item *an_item;
  20.     len = strlen(characters);
  22.     /* construct a list */
  23.     mk_list_init(&items);
  25.     for (i = 0; i < len; i++) {
  26.         an_item = flb_malloc(sizeof(struct item));
  27.         if (!an_item) {
  28.             flb_errno();
  29.             return -1;
  30.         }
  31.         an_item->some_data = characters[i];
  32.         mk_list_add(&an_item->_head, &items);
  33.     }
  35.     /* iterate through the list */
  36.     flb_info("Iterating through list");
  37.     mk_list_foreach_safe(head, tmp, &items) {
  38.         an_item = mk_list_entry(head, struct item, _head);
  39.         flb_info("list item data value: %c", an_item->some_data);
  40.     }
  42.     /* remove an item */
  43.     mk_list_foreach_safe(head, tmp, &items) {
  44.         an_item = mk_list_entry(head, struct item, _head);
  45.         if (an_item->some_data == 'b') {
  46.             mk_list_del(&an_item->_head);
  47.             flb_free(an_item);
  48.         }
  49.     }
  50. }

Message Pack

Fluent Bit uses msgpack to internally store data. If you write code for Fluent Bit, it is almost certain that you will interact with msgpack.

Fluent Bit embeds the msgpack-c library. The example below shows manipulating message pack to add a new key-value pair to a record. In Fluent Bit, the filter_record_modifier plugin adds or deletes keys from records. See its code for more.

  1. #define A_NEW_KEY        "key"
  2. #define A_NEW_KEY_LEN    3
  3. #define A_NEW_VALUE      "value"
  4. #define A_NEW_VALUE_LEN  5
  6. static int cb_filter(const void *data, size_t bytes,
  7.                      const char *tag, int tag_len,
  8.                      void **out_buf, size_t *out_size,
  9.                      struct flb_filter_instance *f_ins,
  10.                      void *context,
  11.                      struct flb_config *config)
  12. {
  13.     (void) f_ins;
  14.     (void) config;
  15.     size_t off = 0;
  16.     int i = 0;
  17.     int ret;
  18.     struct flb_time tm;
  19.     int total_records;
  20.     int new_keys = 1;
  21.     msgpack_sbuffer tmp_sbuf;
  22.     msgpack_packer tmp_pck;
  23.     msgpack_unpacked result;
  24.     msgpack_object  *obj;
  25.     msgpack_object_kv *kv;
  27.     /* Create temporary msgpack buffer */
  28.     msgpack_sbuffer_init(&tmp_sbuf);
  29.     msgpack_packer_init(&tmp_pck, &tmp_sbuf, msgpack_sbuffer_write);
  31.     /* Iterate over each item */
  32.     msgpack_unpacked_init(&result);
  33.     while (msgpack_unpack_next(&result, data, bytes, &off) == MSGPACK_UNPACK_SUCCESS) {
  34.         /*
  35.          * Each record is a msgpack array [timestamp, map] of the
  36.          * timestamp and record map. We 'unpack' each record, and then re-pack
  37.          * it with the new fields added.
  38.          */
  40.         if (result.data.type != MSGPACK_OBJECT_ARRAY) {
  41.             continue;
  42.         }
  44.         /* unpack the array of [timestamp, map] */
  45.         flb_time_pop_from_msgpack(&tm, &result, &obj);
  47.         /* obj should now be the record map */
  48.         if (obj->type != MSGPACK_OBJECT_MAP) {
  49.             continue;
  50.         }
  52.         /* re-pack the array into a new buffer */
  53.         msgpack_pack_array(&tmp_pck, 2);
  54.         flb_time_append_to_msgpack(&tm, &tmp_pck, 0);
  56.         /* new record map size is old size + the new keys we will add */
  57.         total_records = obj->via.map.size + new_keys;
  58.         msgpack_pack_map(&tmp_pck, total_records);
  60.         /* iterate through the old record map and add it to the new buffer */
  61.         kv = obj->via.map.ptr;
  62.         for(i=0; i < obj->via.map.size; i++) {
  63.             msgpack_pack_object(&tmp_pck, (kv+i)->key);
  64.             msgpack_pack_object(&tmp_pck, (kv+i)->val);
  65.         }
  67.         /* append new keys */
  68.         msgpack_pack_str(&tmp_pck, A_NEW_KEY_LEN);
  69.         msgpack_pack_str_body(&tmp_pck, A_NEW_KEY, A_NEW_KEY_LEN);
  70.         msgpack_pack_str(&tmp_pck, A_NEW_VALUE_LEN);
  71.         msgpack_pack_str_body(&tmp_pck, A_NEW_VALUE, A_NEW_VALUE_LEN);
  73.     }
  74.     msgpack_unpacked_destroy(&result);
  76.     /* link new buffers */
  77.     *out_buf  = tmp_sbuf.data;
  78.     *out_size = tmp_sbuf.size;
  79.     return FLB_FILTER_MODIFIED;

Please also check out the message pack examples on the msgpack-c GitHub repo.

Plugin API

Each plugin is a shared object which is loaded into Fluent Bit using dlopen and dlsym.

Input

The input plugin structure is defined in flb_input.h. There are a number of functions which a plugin can implement, most only implement cb_init, cb_collect, and cb_exit.

The "dummy" input plugin very simple and is an excellent example to review to understand more.

Filter

The structure for filter plugins is defined in flb_filter.h. Each plugin must implement cb_init, cb_filter, and cb_exit.

The filter_record_modifier is a good example of a filter plugin.

Note that filter plugins can not asynchronously make HTTP requests. If your plugin needs to make a request, add the following code when you initialize your flb_upstream:

  1. /* Remove async flag from upstream */
  2. upstream->flags &= ~(FLB_IO_ASYNC);

Output

Output plugins are defined in flb_output.h. Each plugin must implement cb_init, cb_flush, and cb_exit.

The stdout plugin is very simple; review its code to understand how output plugins work.

Testing

During development, you can build Fluent Bit as follows:

  1. cd build
  2. cmake -DFLB_DEV=On ../
  3. make

Note that Fluent Bit uses Cmake 3 and on some systems you may need to invoke it as cmake3.

To enable the unit tests run:

  1. cmake -DFLB_DEV=On -DFLB_TESTS_RUNTIME=On -DFLB_TESTS_INTERNAL=On ../
  2. make

Internal tests are for the internal libraries of Fluent Bit. Runtime tests are for the plugins.

You can run the unit tests with make test, however, this is inconvenient in practice. Each test file will create an executable in the build/bin directory which you can run directly. For example, if you want to run the SDS tests, you can invoke them as follows:

  1. $ ./bin/flb-it-sds
  2. Test sds_usage...                               [   OK   ]
  3. Test sds_printf...                              [   OK   ]
  4. SUCCESS: All unit tests have passed.

Need more help?

The best way to learn how Fluent Bit code works is to read it. If you need help understanding the code, reach out to the community, or open a PR with changes that are a work in progress.