Record Modifier

The Record Modifier Filter plugin allows to append fields or to exclude specific fields.

Configuration Parameters

The plugin supports the following configuration parameters: Remove_key and Allowlist_key are exclusive.

Key Description
Record Append fields. This parameter needs key and value pair.
Remove_key If the key is matched, that field is removed.
Allowlist_key If the key is not matched, that field is removed.
Whitelist_key An alias of Allowlist_key for backwards compatibility.

Getting Started

In order to start filtering records, you can run the filter from the command line or through the configuration file.

This is a sample in_mem record to filter.

  1. {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>299352, "Swap.total"=>2064380, "Swap.used"=>32656, "Swap.free"=>2031724}

Append fields

The following configuration file is to append product name and hostname (via environment variable) to record.

  1. [INPUT]
  2.     Name mem
  3.     Tag  mem.local
  5. [OUTPUT]
  6.     Name  stdout
  7.     Match *
  9. [FILTER]
  10.     Name record_modifier
  11.     Match *
  12.     Record hostname ${HOSTNAME}
  13.     Record product Awesome_Tool

You can also run the filter from command line.

  1. $ fluent-bit -i mem -o stdout -F record_modifier -p 'Record=hostname ${HOSTNAME}' -p 'Record=product Awesome_Tool' -m '*'

The output will be

  1. [0] mem.local: [1492436882.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>299352, "Swap.total"=>2064380, "Swap.used"=>32656, "Swap.free"=>2031724, "hostname"=>"localhost.localdomain", "product"=>"Awesome_Tool"}]

Remove fields with Remove_key

The following configuration file is to remove ‘Swap.*‘ fields.

  1. [INPUT]
  2.     Name mem
  3.     Tag  mem.local
  5. [OUTPUT]
  6.     Name  stdout
  7.     Match *
  9. [FILTER]
  10.     Name record_modifier
  11.     Match *
  12.     Remove_key Swap.total
  13.     Remove_key Swap.used
  14.     Remove_key Swap.free

You can also run the filter from command line.

  1. $ fluent-bit -i mem -o stdout -F  record_modifier -p 'Remove_key=Swap.total' -p 'Remove_key=Swap.free' -p 'Remove_key=Swap.used' -m '*'

The output will be

  1. [0] mem.local: [1492436998.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>295332}]

Remove fields with Allowlist_key

The following configuration file is to remain ‘Mem.*‘ fields.

  1. [INPUT]
  2.     Name mem
  3.     Tag  mem.local
  5. [OUTPUT]
  6.     Name  stdout
  7.     Match *
  9. [FILTER]
  10.     Name record_modifier
  11.     Match *
  12.     Allowlist_key Mem.total
  13.     Allowlist_key Mem.used
  14.     Allowlist_key Mem.free

You can also run the filter from command line.

  1. $ fluent-bit -i mem -o stdout -F  record_modifier -p 'Allowlist_key=Mem.total' -p 'Allowlist_key=Mem.free' -p 'Allowlist_key=Mem.used' -m '*'

The output will be

  1. [0] mem.local: [1492436998.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>295332}]