Record Modifier

The Record Modifier Filter plugin allows to append fields or to exclude specific fields.

Configuration Parameters

The plugin supports the following configuration parameters: Remove_key and Whitelist_key are exclusive.

Key Description
Record Append fields. This parameter needs key and value pair.
Remove_key If the key is matched, that field is removed.
Whitelist_key If the key is not matched, that field is removed.

Getting Started

In order to start filtering records, you can run the filter from the command line or through the configuration file.

This is a sample in_mem record to filter.

  1. {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>299352, "Swap.total"=>2064380, "Swap.used"=>32656, "Swap.free"=>2031724}

Append fields

The following configuration file is to append product name and hostname (via environment variable) to record.

  1. [INPUT]
  2.     Name mem
  3.     Tag  mem.local
  5. [OUTPUT]
  6.     Name  stdout
  7.     Match *
  9. [FILTER]
  10.     Name record_modifier
  11.     Match *
  12.     Record hostname ${HOSTNAME}
  13.     Record product Awesome_Tool

You can also run the filter from command line.

  1. $ fluent-bit -i mem -o stdout -F record_modifier -p 'Record=hostname ${HOSTNAME}' -p 'Record=product Awesome_Tool' -m '*'

The output will be

  1. [0] mem.local: [1492436882.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>299352, "Swap.total"=>2064380, "Swap.used"=>32656, "Swap.free"=>2031724, "hostname"=>"localhost.localdomain", "product"=>"Awesome_Tool"}]

Remove fields with Remove_key

The following configuration file is to remove ‘Swap.*‘ fields.

  1. [INPUT]
  2.     Name mem
  3.     Tag  mem.local
  5. [OUTPUT]
  6.     Name  stdout
  7.     Match *
  9. [FILTER]
  10.     Name record_modifier
  11.     Match *
  12.     Remove_key Swap.total
  13.     Remove_key Swap.used
  14.     Remove_key Swap.free

You can also run the filter from command line.

  1. $ fluent-bit -i mem -o stdout -F  record_modifier -p 'Remove_key=Swap.total' -p 'Remove_key=Swap.free' -p 'Remove_key=Swap.used' -m '*'

The output will be

  1. [0] mem.local: [1492436998.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>295332}]

Remove fields with Whitelist_key

The following configuration file is to remain ‘Mem.*‘ fields.

  1. [INPUT]
  2.     Name mem
  3.     Tag  mem.local
  5. [OUTPUT]
  6.     Name  stdout
  7.     Match *
  9. [FILTER]
  10.     Name record_modifier
  11.     Match *
  12.     Whitelist_key Mem.total
  13.     Whitelist_key Mem.used
  14.     Whitelist_key Mem.free

You can also run the filter from command line.

  1. $ fluent-bit -i mem -o stdout -F  record_modifier -p 'Whitelist_key=Mem.total' -p 'Whitelist_key=Mem.free' -p 'Whitelist_key=Mem.used' -m '*'

The output will be

  1. [0] mem.local: [1492436998.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>295332}]