description: >- Concatenate Multiline or Stack trace log messages. Available on Fluent Bit >= v1.8.2.

Multiline

The Multiline Filter helps to concatenate messages that originally belong to one context but were split across multiple records or log lines. Common examples are stack traces or applications that print logs in multiple lines.

As part of the built-in functionality, without major configuration effort, you can enable one of ours built-in parsers with auto detection and multi format support:

  • go
  • python
  • ruby
  • java (Google Cloud Platform Java stacktrace format)

Some comments about this filter:

  • The usage of this filter depends on a previous configuration of a Multiline Parser definition.
  • If you aim to concatenate messages split originally by Docker or CRI container engines, we recommend doing the concatenation on Tail plugin, this same functionality exists there.

Configuration Parameters

The plugin supports the following configuration parameters:

Property Description
multiline.parser Specify one or multiple Multiline Parser definitions to apply to the content. You can specify multiple multiline parsers to detect different formats by separating them with a comma.
multiline.key_content Key name that holds the content to process. Note that a Multiline Parser definition can already specify the key_content to use, but this option allows to overwrite that value for the purpose of the filter.

Configuration Example

The following example aims to parse a log file called test.log that contains some full lines, a custom Java stacktrace and a Go stacktrace.

{% hint style=”info” %} The following example files can be located at:

https://github.com/fluent/fluent-bit/tree/master/documentation/examples/multiline/filter_multiline {% endhint %}

Example files content:

{% tabs %} {% tab title=”fluent-bit.conf” %} This is the primary Fluent Bit configuration file. It includes the parsers_multiline.conf and tails the file test.log by applying the multiline parsers multiline-regex-test and go. Then it sends the processing to the standard output. 

  1. [SERVICE]
  2.     flush                 1
  3.     log_level             info
  4.     parsers_file          parsers_multiline.conf
  6. [INPUT]
  7.     name                  tail
  8.     path                  test.log
  9.     read_from_head        true
  11. [FILTER]
  12.     name                  multiline
  13.     match                 *
  14.     multiline.key_content log
  15.     multiline.parser      go, multiline-regex-test
  17. [OUTPUT]
  18.     name                  stdout
  19.     match                 *

{% endtab %}

{% tab title=”parsers_multiline.conf” %} This second file defines a multiline parser for the example. Note that a second multiline parser called go is used in fluent-bit.conf, but this one is a built-in parser.

  1. [MULTILINE_PARSER]
  2.     name          multiline-regex-test
  3.     type          regex
  4.     flush_timeout 1000
  5.     #
  6.     # Regex rules for multiline parsing
  7.     # ---------------------------------
  8.     #
  9.     # configuration hints:
  10.     #
  11.     #  - first state always has the name: start_state
  12.     #  - every field in the rule must be inside double quotes
  13.     #
  14.     # rules |   state name  | regex pattern                  | next state
  15.     # ------|---------------|--------------------------------------------
  16.     rule      "start_state"   "/(Dec \d+ \d+\:\d+\:\d+)(.*)/"  "cont"
  17.     rule      "cont"          "/^\s+at.*/"                     "cont"

{% endtab %}

{% tab title=”test.log” %} An example file with multiline and multiformat content:

  1. single line...
  2. Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
  3.     at com.myproject.module.MyProject.badMethod(MyProject.java:22)
  4.     at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
  5.     at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
  6.     at com.myproject.module.MyProject.someMethod(MyProject.java:10)
  7.     at com.myproject.module.MyProject.main(MyProject.java:6)
  8. another line...
  9. panic: my panic
  11. goroutine 4 [running]:
  12. panic(0x45cb40, 0x47ad70)
  13.   /usr/local/go/src/runtime/panic.go:542 +0x46c fp=0xc42003f7b8 sp=0xc42003f710 pc=0x422f7c
  14. main.main.func1(0xc420024120)
  15.   foo.go:6 +0x39 fp=0xc42003f7d8 sp=0xc42003f7b8 pc=0x451339
  16. runtime.goexit()
  17.   /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003f7e0 sp=0xc42003f7d8 pc=0x44b4d1
  18. created by main.main
  19.   foo.go:5 +0x58
  21. goroutine 1 [chan receive]:
  22. runtime.gopark(0x4739b8, 0xc420024178, 0x46fcd7, 0xc, 0xc420028e17, 0x3)
  23.   /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc420053e30 sp=0xc420053e00 pc=0x42503c
  24. runtime.goparkunlock(0xc420024178, 0x46fcd7, 0xc, 0x1000f010040c217, 0x3)
  25.   /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc420053e70 sp=0xc420053e30 pc=0x42512e
  26. runtime.chanrecv(0xc420024120, 0x0, 0xc420053f01, 0x4512d8)
  27.   /usr/local/go/src/runtime/chan.go:506 +0x304 fp=0xc420053f20 sp=0xc420053e70 pc=0x4046b4
  28. runtime.chanrecv1(0xc420024120, 0x0)
  29.   /usr/local/go/src/runtime/chan.go:388 +0x2b fp=0xc420053f50 sp=0xc420053f20 pc=0x40439b
  30. main.main()
  31.   foo.go:9 +0x6f fp=0xc420053f80 sp=0xc420053f50 pc=0x4512ef
  32. runtime.main()
  33.   /usr/local/go/src/runtime/proc.go:185 +0x20d fp=0xc420053fe0 sp=0xc420053f80 pc=0x424bad
  34. runtime.goexit()
  35.   /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420053fe8 sp=0xc420053fe0 pc=0x44b4d1
  37. goroutine 2 [force gc (idle)]:
  38. runtime.gopark(0x4739b8, 0x4ad720, 0x47001e, 0xf, 0x14, 0x1)
  39.   /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc42003e768 sp=0xc42003e738 pc=0x42503c
  40. runtime.goparkunlock(0x4ad720, 0x47001e, 0xf, 0xc420000114, 0x1)
  41.   /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc42003e7a8 sp=0xc42003e768 pc=0x42512e
  42. runtime.forcegchelper()
  43.   /usr/local/go/src/runtime/proc.go:238 +0xcc fp=0xc42003e7e0 sp=0xc42003e7a8 pc=0x424e5c
  44. runtime.goexit()
  45.   /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003e7e8 sp=0xc42003e7e0 pc=0x44b4d1
  46. created by runtime.init.4
  47.   /usr/local/go/src/runtime/proc.go:227 +0x35
  49. goroutine 3 [GC sweep wait]:
  50. runtime.gopark(0x4739b8, 0x4ad7e0, 0x46fdd2, 0xd, 0x419914, 0x1)
  51.   /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc42003ef60 sp=0xc42003ef30 pc=0x42503c
  52. runtime.goparkunlock(0x4ad7e0, 0x46fdd2, 0xd, 0x14, 0x1)
  53.   /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc42003efa0 sp=0xc42003ef60 pc=0x42512e
  54. runtime.bgsweep(0xc42001e150)
  55.   /usr/local/go/src/runtime/mgcsweep.go:52 +0xa3 fp=0xc42003efd8 sp=0xc42003efa0 pc=0x419973
  56. runtime.goexit()
  57.   /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003efe0 sp=0xc42003efd8 pc=0x44b4d1
  58. created by runtime.gcenable
  59.   /usr/local/go/src/runtime/mgc.go:216 +0x58
  60. one more line, no multiline

{% endtab %} {% endtabs %}

By running Fluent Bit with the given configuration file you will obtain:

  1. $ fluent-bit -c fluent-bit.conf 
  3. [0] tail.0: [1626736433.143567481, {"log"=>"single line..."}]
  4. [1] tail.0: [1626736433.143570538, {"log"=>"Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
  5.     at com.myproject.module.MyProject.badMethod(MyProject.java:22)
  6.     at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
  7.     at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
  8.     at com.myproject.module.MyProject.someMethod(MyProject.java:10)
  9.     at com.myproject.module.MyProject.main(MyProject.java:6)"}]
  10. [2] tail.0: [1626736433.143572538, {"log"=>"another line..."}]
  11. [3] tail.0: [1626736433.143572894, {"log"=>"panic: my panic
  13. goroutine 4 [running]:
  14. panic(0x45cb40, 0x47ad70)
  15.   /usr/local/go/src/runtime/panic.go:542 +0x46c fp=0xc42003f7b8 sp=0xc42003f710 pc=0x422f7c
  16. main.main.func1(0xc420024120)
  17.   foo.go:6 +0x39 fp=0xc42003f7d8 sp=0xc42003f7b8 pc=0x451339
  18. runtime.goexit()
  19.   /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003f7e0 sp=0xc42003f7d8 pc=0x44b4d1
  20. created by main.main
  21.   foo.go:5 +0x58
  23. goroutine 1 [chan receive]:
  24. runtime.gopark(0x4739b8, 0xc420024178, 0x46fcd7, 0xc, 0xc420028e17, 0x3)
  25.   /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc420053e30 sp=0xc420053e00 pc=0x42503c
  26. runtime.goparkunlock(0xc420024178, 0x46fcd7, 0xc, 0x1000f010040c217, 0x3)
  27.   /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc420053e70 sp=0xc420053e30 pc=0x42512e
  28. runtime.chanrecv(0xc420024120, 0x0, 0xc420053f01, 0x4512d8)
  29.   /usr/local/go/src/runtime/chan.go:506 +0x304 fp=0xc420053f20 sp=0xc420053e70 pc=0x4046b4
  30. runtime.chanrecv1(0xc420024120, 0x0)
  31.   /usr/local/go/src/runtime/chan.go:388 +0x2b fp=0xc420053f50 sp=0xc420053f20 pc=0x40439b
  32. main.main()
  33.   foo.go:9 +0x6f fp=0xc420053f80 sp=0xc420053f50 pc=0x4512ef
  34. runtime.main()
  35.   /usr/local/go/src/runtime/proc.go:185 +0x20d fp=0xc420053fe0 sp=0xc420053f80 pc=0x424bad
  36. runtime.goexit()
  37.   /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420053fe8 sp=0xc420053fe0 pc=0x44b4d1
  39. goroutine 2 [force gc (idle)]:
  40. runtime.gopark(0x4739b8, 0x4ad720, 0x47001e, 0xf, 0x14, 0x1)
  41.   /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc42003e768 sp=0xc42003e738 pc=0x42503c
  42. runtime.goparkunlock(0x4ad720, 0x47001e, 0xf, 0xc420000114, 0x1)
  43.   /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc42003e7a8 sp=0xc42003e768 pc=0x42512e
  44. runtime.forcegchelper()
  45.   /usr/local/go/src/runtime/proc.go:238 +0xcc fp=0xc42003e7e0 sp=0xc42003e7a8 pc=0x424e5c
  46. runtime.goexit()
  47.   /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003e7e8 sp=0xc42003e7e0 pc=0x44b4d1
  48. created by runtime.init.4
  49.   /usr/local/go/src/runtime/proc.go:227 +0x35
  51. goroutine 3 [GC sweep wait]:
  52. runtime.gopark(0x4739b8, 0x4ad7e0, 0x46fdd2, 0xd, 0x419914, 0x1)
  53.   /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc42003ef60 sp=0xc42003ef30 pc=0x42503c
  54. runtime.goparkunlock(0x4ad7e0, 0x46fdd2, 0xd, 0x14, 0x1)
  55.   /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc42003efa0 sp=0xc42003ef60 pc=0x42512e
  56. runtime.bgsweep(0xc42001e150)
  57.   /usr/local/go/src/runtime/mgcsweep.go:52 +0xa3 fp=0xc42003efd8 sp=0xc42003efa0 pc=0x419973
  58. runtime.goexit()
  59.   /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003efe0 sp=0xc42003efd8 pc=0x44b4d1
  60. created by runtime.gcenable
  61.   /usr/local/go/src/runtime/mgc.go:216 +0x58"}]
  62. [4] tail.0: [1626736433.143585473, {"log"=>"one more line, no multiline"}]

The lines that did not match a pattern are not considered as part of the multiline message, while the ones that matched the rules were concatenated properly.