Modify

The Modify Filter plugin allows you to change records using rules and conditions.

Example usage

As an example using JSON notation to,

  • Rename Key2 to RenamedKey
  • Add a key OtherKey with value Value3 if OtherKey does not yet exist

Example (input)

  1. {
  2.   "Key1"     : "Value1",
  3.   "Key2"     : "Value2"
  4. }

Example (output)

  1. {
  2.   "Key1"       : "Value1",
  3.   "RenamedKey" : "Value2",
  4.   "OtherKey"   : "Value3"
  5. }

Configuration Parameters

Rules

The plugin supports the following rules:

Operation Parameter 1 Parameter 2 Description
Set STRING:KEY STRING:VALUE Add a key/value pair with key KEY and value VALUE. If KEY already exists, this field is overwritten
Add STRING:KEY STRING:VALUE Add a key/value pair with key KEY and value VALUE if KEY does not exist
Remove STRING:KEY NONE Remove a key/value pair with key KEY if it exists
Remove_wildcard WILDCARD:KEY NONE Remove all key/value pairs with key matching wildcard KEY
Remove_regex REGEXP:KEY NONE Remove all key/value pairs with key matching regexp KEY
Rename STRING:KEY STRING:RENAMED_KEY Rename a key/value pair with key KEY to RENAMED_KEY if KEY exists AND RENAMED_KEY does not exist
Hard_rename STRING:KEY STRING:RENAMED_KEY Rename a key/value pair with key KEY to RENAMED_KEY if KEY exists. If RENAMED_KEY already exists, this field is overwritten
Copy STRING:KEY STRING:COPIED_KEY Copy a key/value pair with key KEY to COPIED_KEY if KEY exists AND COPIED_KEY does not exist
Hard_copy STRING:KEY STRING:COPIED_KEY Copy a key/value pair with key KEY to COPIED_KEY if KEY exists. If COPIED_KEY already exists, this field is overwritten
  • Rules are case insensitive, parameters are not
  • Any number of rules can be set in a filter instance.
  • Rules are applied in the order they appear, with each rule operating on the result of the previous rule.

Conditions

The plugin supports the following conditions:

Condition Parameter Parameter 2 Description
Key_exists STRING:KEY NONE Is true if KEY exists
Key_does_not_exist STRING:KEY STRING:VALUE Is true if KEY does not exist
A_key_matches REGEXP:KEY NONE Is true if a key matches regex KEY
No_key_matches REGEXP:KEY NONE Is true if no key matches regex KEY
Key_value_equals STRING:KEY STRING:VALUE Is true if KEY exists and its value is VALUE
Key_value_does_not_equal STRING:KEY STRING:VALUE Is true if KEY exists and its value is not VALUE
Key_value_matches STRING:KEY REGEXP:VALUE Is true if key KEY exists and its value matches VALUE
Key_value_does_not_match STRING:KEY REGEXP:VALUE Is true if key KEY exists and its value does not match VALUE
Matching_keys_have_matching_values REGEXP:KEY REGEXP:VALUE Is true if all keys matching KEY have values that match VALUE
Matching_keys_do_not_have_matching_values REGEXP:KEY REGEXP:VALUE Is true if all keys matching KEY have values that do not match VALUE
  • Conditions are case insensitive, parameters are not
  • Any number of conditions can be set.
  • Conditions apply to the whole filter instance and all its rules. Not to individual rules.
  • All conditions have to be true for the rules to be applied.

Example #1 - Add and Rename

In order to start filtering records, you can run the filter from the command line or through the configuration file. The following invokes the Memory Usage Input Plugin, which outputs the following (example),

  1. [0] memory: [1488543156, {"Mem.total"=>1016044, "Mem.used"=>841388, "Mem.free"=>174656, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
  2. [1] memory: [1488543157, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
  3. [2] memory: [1488543158, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
  4. [3] memory: [1488543159, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]

Using command Line

Note: Using the command line mode requires quotes parse the wildcard properly. The use of a configuration file is recommended.

  1. bin/fluent-bit -i mem \
  2.   -p 'tag=mem.local' \
  3.   -F modify \
  4.   -p 'Add=Service1 SOMEVALUE' \
  5.   -p 'Add=Service2 SOMEVALUE3' \
  6.   -p 'Add=Mem.total2 TOTALMEM2' \
  7.   -p 'Rename=Mem.free MEMFREE' \
  8.   -p 'Rename=Mem.used MEMUSED' \
  9.   -p 'Rename=Swap.total SWAPTOTAL' \
  10.   -p 'Add=Mem.total TOTALMEM' \
  11.   -m '*' \
  12.   -o stdout

Configuration File

  1. [INPUT]
  2.     Name mem
  3.     Tag  mem.local
  5. [OUTPUT]
  6.     Name  stdout
  7.     Match *
  9. [FILTER]
  10.     Name modify
  11.     Match *
  12.     Add Service1 SOMEVALUE
  13.     Add Service3 SOMEVALUE3
  14.     Add Mem.total2 TOTALMEM2
  15.     Rename Mem.free MEMFREE
  16.     Rename Mem.used MEMUSED
  17.     Rename Swap.total SWAPTOTAL
  18.     Add Mem.total TOTALMEM

Result

The output of both the command line and configuration invocations should be identical and result in the following output.

  1. [2018/04/06 01:35:13] [ info] [engine] started
  2. [0] mem.local: [1522980610.006892802, {"Mem.total"=>4050908, "MEMUSED"=>738100, "MEMFREE"=>3312808, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}]
  3. [1] mem.local: [1522980611.000658288, {"Mem.total"=>4050908, "MEMUSED"=>738068, "MEMFREE"=>3312840, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}]
  4. [2] mem.local: [1522980612.000307652, {"Mem.total"=>4050908, "MEMUSED"=>738068, "MEMFREE"=>3312840, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}]
  5. [3] mem.local: [1522980613.000122671, {"Mem.total"=>4050908, "MEMUSED"=>738068, "MEMFREE"=>3312840, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}]

Example #2 - Conditionally Add and Remove

Configuration File

  1. [INPUT]
  2.     Name mem
  3.     Tag  mem.local
  4.     Interval_Sec 1
  6. [FILTER]
  7.     Name    modify
  8.     Match   mem.*
  10.     Condition Key_Does_Not_Exist cpustats
  11.     Condition Key_Exists Mem.used
  13.     Set cpustats UNKNOWN
  15. [FILTER]
  16.     Name    modify
  17.     Match   mem.*
  19.     Condition Key_Value_Does_Not_Equal cpustats KNOWN
  21.     Add sourcetype memstats
  23. [FILTER]
  24.     Name    modify
  25.     Match   mem.*
  27.     Condition Key_Value_Equals cpustats UNKNOWN
  29.     Remove_wildcard Mem
  30.     Remove_wildcard Swap
  31.     Add cpustats_more STILL_UNKNOWN
  33. [OUTPUT]
  34.     Name           stdout
  35.     Match          *

Result

  1. [2018/06/14 07:37:34] [ info] [engine] started (pid=1493)
  2. [0] mem.local: [1528925855.000223110, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}]
  3. [1] mem.local: [1528925856.000064516, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}]
  4. [2] mem.local: [1528925857.000165965, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}]
  5. [3] mem.local: [1528925858.000152319, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}]

Example #3 - Emoji

Configuration File

  1. [INPUT]
  2.     Name mem
  3.     Tag  mem.local
  5. [OUTPUT]
  6.     Name  stdout
  7.     Match *
  9. [FILTER]
  10.     Name modify
  11.     Match *
  13.     Remove_Wildcard Mem
  14.     Remove_Wildcard Swap
  15.     Set This_plugin_is_on 🔥
  16.     Set 🔥 is_hot
  17.     Copy 🔥 💦
  18.     Rename  💦 ❄️
  19.     Set ❄️ is_cold
  20.     Set 💦 is_wet

Result

  1. [2018/06/14 07:46:11] [ info] [engine] started (pid=21875)
  2. [0] mem.local: [1528926372.000197916, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]
  3. [1] mem.local: [1528926373.000107868, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]
  4. [2] mem.local: [1528926374.000181042, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]
  5. [3] mem.local: [1528926375.000090841, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]
  6. [0] mem.local: [1528926376.000610974, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]