Nest

The Nest Filter plugin allows you to operate on or with nested data. Its modes of operation are

  • nest - Take a set of records and place them in a map
  • lift - Take a map by key and lift its records up

Example usage (nest)

As an example using JSON notation, to nest keys matching the Wildcard value Key* under a new key NestKey the transformation becomes,

Example (input)

  1. {
  2.   "Key1"     : "Value1",
  3.   "Key2"     : "Value2",
  4.   "OtherKey" : "Value3"
  5. }

Example (output)

  1. {
  2.   "OtherKey" : "Value3"
  3.   "NestKey"  : {
  4.     "Key1"     : "Value1",
  5.     "Key2"     : "Value2",
  6.   }
  7. }

Example usage (lift)

As an example using JSON notation, to lift keys nested under the Nested_under value NestKey* the transformation becomes,

Example (input)

  1. {
  2.   "OtherKey" : "Value3"
  3.   "NestKey"  : {
  4.     "Key1"     : "Value1",
  5.     "Key2"     : "Value2",
  6.   }
  7. }

Example (output)

  1. {
  2.   "Key1"     : "Value1",
  3.   "Key2"     : "Value2",
  4.   "OtherKey" : "Value3"
  5. }

Configuration Parameters

The plugin supports the following configuration parameters:

Key Value Format Operation Description
Operation ENUM [nest or lift] Select the operation nest or lift
Wildcard FIELD WILDCARD nest Nest records which field matches the wildcard
Nest_under FIELD STRING nest Nest records matching the Wildcard under this key
Nested_under FIELD STRING lift Lift records nested under the Nested_under key
Add_prefix FIELD STRING ANY Prefix affected keys with this string
Remove_prefix FIELD STRING ANY Remove prefix from affected keys if it matches this string

Getting Started

In order to start filtering records, you can run the filter from the command line or through the configuration file. The following invokes the Memory Usage Input Plugin, which outputs the following (example),

  1. [0] memory: [1488543156, {"Mem.total"=>1016044, "Mem.used"=>841388, "Mem.free"=>174656, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]

Example #1 - nest

Command Line

Note: Using the command line mode requires quotes parse the wildcard properly. The use of a configuration file is recommended.

The following command will load the mem plugin. Then the nest filter will match the wildcard rule to the keys and nest the keys matching Mem.* under the new key NEST.

  1. $ bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildcard=Mem.*' -p 'Nest_under=Memstats' -p 'Remove_prefix=Mem.' -m '*' -o stdout

Configuration File

  1. [INPUT]
  2.     Name mem
  3.     Tag  mem.local
  5. [OUTPUT]
  6.     Name  stdout
  7.     Match *
  9. [FILTER]
  10.     Name nest
  11.     Match *
  12.     Operation nest
  13.     Wildcard Mem.*
  14.     Nest_under Memstats
  15.     Remove_prefix Mem.

Result

The output of both the command line and configuration invocations should be identical and result in the following output.

  1. [2018/04/06 01:35:13] [ info] [engine] started
  2. [0] mem.local: [1522978514.007359767, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Memstats"=>{"total"=>4050908, "used"=>714984, "free"=>3335924}}]

Example #1 - nest and lift undo

This example nests all Mem.* and Swap,* items under the Stats key and then reverses these actions with a lift operation. The output appears unchanged.

Configuration File

  1. [INPUT]
  2.     Name mem
  3.     Tag  mem.local
  5. [OUTPUT]
  6.     Name  stdout
  7.     Match *
  9. [FILTER]
  10.     Name nest
  11.     Match *
  12.     Operation nest
  13.     Wildcard Mem.*
  14.     Wildcard Swap.*
  15.     Nest_under Stats
  16.     Add_prefix NESTED
  18. [FILTER]
  19.     Name nest
  20.     Match *
  21.     Operation lift
  22.     Nested_under Stats
  23.     Remove_prefix NESTED

Result

  1. [2018/06/21 17:42:37] [ info] [engine] started (pid=17285)
  2. [0] mem.local: [1529566958.000940636, {"Mem.total"=>8053656, "Mem.used"=>6940380, "Mem.free"=>1113276, "Swap.total"=>16532988, "Swap.used"=>1286772, "Swap.free"=>15246216}]

Example #2 - nest 3 levels deep

This example takes the keys starting with Mem.* and nests them under LAYER1, which itself is then nested under LAYER2, which is nested under LAYER3.

Configuration File

  1. [INPUT]
  2.     Name mem
  3.     Tag  mem.local
  5. [OUTPUT]
  6.     Name  stdout
  7.     Match *
  9. [FILTER]
  10.     Name nest
  11.     Match *
  12.     Operation nest
  13.     Wildcard Mem.*
  14.     Nest_under LAYER1
  16. [FILTER]
  17.     Name nest
  18.     Match *
  19.     Operation nest
  20.     Wildcard LAYER1*
  21.     Nest_under LAYER2
  23. [FILTER]
  24.     Name nest
  25.     Match *
  26.     Operation nest
  27.     Wildcard LAYER2*
  28.     Nest_under LAYER3

Result

  1. [0] mem.local: [1524795923.009867831, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "LAYER3"=>{"LAYER2"=>{"LAYER1"=>{"Mem.total"=>4050908, "Mem.used"=>1112036, "Mem.free"=>2938872}}}}]
  4. {
  5.   "Swap.total"=>1046524,
  6.   "Swap.used"=>0,
  7.   "Swap.free"=>1046524,
  8.   "LAYER3"=>{
  9.     "LAYER2"=>{
  10.       "LAYER1"=>{
  11.         "Mem.total"=>4050908,
  12.         "Mem.used"=>1112036,
  13.         "Mem.free"=>2938872
  14.       }
  15.     }
  16.   }
  17. }

Example #3 - multiple nest and lift filters with prefix

This example starts with the 3-level deep nesting of Example 2 and applies the lift filter three times to reverse the operations. The end result is that all records are at the top level, without nesting, again. One prefix is added for each level that is lifted.

Configuration file

  1. [INPUT]
  2.     Name mem
  3.     Tag  mem.local
  5. [OUTPUT]
  6.     Name  stdout
  7.     Match *
  9. [FILTER]
  10.     Name nest
  11.     Match *
  12.     Operation nest
  13.     Wildcard Mem.*
  14.     Nest_under LAYER1
  16. [FILTER]
  17.     Name nest
  18.     Match *
  19.     Operation nest
  20.     Wildcard LAYER1*
  21.     Nest_under LAYER2
  23. [FILTER]
  24.     Name nest
  25.     Match *
  26.     Operation nest
  27.     Wildcard LAYER2*
  28.     Nest_under LAYER3
  30. [FILTER]
  31.     Name nest
  32.     Match *
  33.     Operation lift
  34.     Nested_under LAYER3
  35.     Add_prefix Lifted3_
  37. [FILTER]
  38.     Name nest
  39.     Match *
  40.     Operation lift
  41.     Nested_under Lifted3_LAYER2
  42.     Add_prefix Lifted3_Lifted2_
  44. [FILTER]
  45.     Name nest
  46.     Match *
  47.     Operation lift
  48.     Nested_under Lifted3_Lifted2_LAYER1
  49.     Add_prefix Lifted3_Lifted2_Lifted1_

Result

  1. [0] mem.local: [1524862951.013414798, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Lifted3_Lifted2_Lifted1_Mem.total"=>4050908, "Lifted3_Lifted2_Lifted1_Mem.used"=>1253912, "Lifted3_Lifted2_Lifted1_Mem.free"=>2796996}]
  4. {
  5.   "Swap.total"=>1046524, 
  6.   "Swap.used"=>0, 
  7.   "Swap.free"=>1046524, 
  8.   "Lifted3_Lifted2_Lifted1_Mem.total"=>4050908, 
  9.   "Lifted3_Lifted2_Lifted1_Mem.used"=>1253912, 
  10.   "Lifted3_Lifted2_Lifted1_Mem.free"=>2796996
  11. }