transformer

Used to perform functional data processing conditional judgement.
Source interceptor.

Use Cases

  • Extracts the log level and drops the DEBUG log.
  • Check whether the log is in json form when logs in both json and plain form are mixed.
  • Add different topic topics according to the status code in log.
  • …

See reference.

Usage

The transformer will execute all actions in order in the configured actions. The action is similar to the function, and parameters can be written. The parameters are generally fields in the event. Each action may also include additional fields. Take the following regex(body) for example, body is the parameter of regex, and pattern is an extra field.

  1. interceptors:
  2.   - type: transformer
  3.     actions:
  4.       - action: regex(body)
  5.         pattern: ^(?P<time>[^ ^Z]+Z) (?P<level>[^ ]*) (?P<log>.*)$
  6.       - action: add(topic, common)

In addition, action also supports if-then-else:

  1. - if: <condition>
  2.   then:
  3.     - action: funcA()
  4.   else:
  5.     - action: funcB()

Condition is also in the form of a function.

  1. interceptors:
  2.   - type: transformer
  3.     actions:
  4.       - if: equal(status, 404)
  5.         then:
  6.           - action: add(topic, not_found)
  7.           - action: return()

action

Common fields

  • ignoreError: Indicates whether to ignore and print errors during the processing of this action.

Example

  1. - type: transformer
  2.   actions:
  3.     - action: regex(body)
  4.       pattern: (?<ip>\S+) (?<id>\S+) (?<u>\S+) (?<time>\[.*?\]) (?<url>\".*?\") (?<status>\S+) (?<size>\S+)
  5.       ignoreError: true

The ignoreError here is set to true, which means that the regular matching error will be ignored, and subsequent actions will continue to be executed.

add(key, value)

Add additional key:value to the event.

Example

  1. - action: add(topic, loggie)

input:

  1. {
  2.   "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body"
  3. }

output:

  1. {
  2.   "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
  3.   "topic": "loggie"
  4. }

copy(from, to)

Copy the fields in the event.

parameter:

  • from: original key
  • to: new key

Example

  1. - action: copy(foo, bar)

input:

  1. {
  2.   "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
  3.   "foo": "loggie"
  4. }

output:

  1. {
  2.   "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
  3.   "foo": "loggie",
  4.   "bar": "loggie"
  5. }

move(from, to)

Move/rename fields.

parameter:

  • from: original key
  • to: new key

Example

  1. - action: move(foo, bar)

input:

  1. {
  2.   "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
  3.   "foo": "loggie"
  4. }

output:

  1. {
  2.   "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
  3.   "bar": "loggie"
  4. }

set(key, value)

Update the value of key to be value.

parameter:

  • key: the field to be updated
  • value: new value

Example

  1. - action: set(foo, test)

input:

  1. {
  2.   "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
  3.   "foo": "loggie"
  4. }

output:

  1. {
  2.   "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
  3.   "foo": "test"
  4. }

del(key1, key2…)

Delete fields. Multiple field keys can be filled in.

Example

  1. - action: del(foo)

input:

  1. {
  2.   "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
  3.   "foo": "loggie"
  4. }

output:

  1. {
  2.   "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
  3. }

underRoot(key)

Put nested fields at the root (outermost).

Example

  1. - action: underRoot(state)

input:

  1. {
  2.   "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
  3.   "state": {
  4.     "node": "127.0.0.1",
  5.     "phase": "running"
  6.   }
  7. }

output:

  1. {
  2.   "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
  3.   "node": "127.0.0.1",
  4.   "phase": "running"
  5. }

fmt(key)

Generate a value based on other field values. If the key does not exist, the field will be added.

Extra fields:

  • pattern: Required, indicating formatting rules, eg ${state.node}-${state.phase}. If pattern is a fixed value, the action is similar to set(key, value).

Example

  1. - action: fmt(status)
  2.   pattern: ${state.node} is ${state.phase}

input:

  1. {
  2.   "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
  3.   "state": {
  4.     "node": "127.0.0.1",
  5.     "phase": "running"
  6.   }
  7. }

output:

  1. {
  2.   "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
  3.   "state": {
  4.     "node": "127.0.0.1",
  5.     "phase": "running"
  6.   }，
  7.   "status": "127.0.0.1 is running"
  8. }

timestamp(key)

Time format conversion.

Extra fields:

  • fromLayout: required, specifies the time format of the field (golang form), and can also be unix and unix_ms.
  • fromLocation: specifies the time zone of the field. UTC and Local also supported. If empty, UTC is used.
  • toLayout: required, the converted time format (golang form), and can also be unix and unix_ms.
  • toLocation: the converted time zone. UTC and Local also supported. If empty, UTC is used.

Example

  1. - action: timestamp(time)
  2.   fromLayout: "2006-01-02 15:04:05"
  3.   fromLocation: Asia/Shanghai
  4.   toLayout: unix_ms
  5.   toLocation: Local

input:

  1. {
  2.   "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
  3.   "time": "2022-06-28 11:24:35"
  4. }

output:

  1. {
  2.   "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
  3.   "time": 1656386675000
  4. }

For time in golang form, please refer to:

  1. const (
  2.     Layout      = "01/02 03:04:05PM '06 -0700" // The reference time, in numerical order.
  3.     ANSIC       = "Mon Jan _2 15:04:05 2006"
  4.     UnixDate    = "Mon Jan _2 15:04:05 MST 2006"
  5.     RubyDate    = "Mon Jan 02 15:04:05 -0700 2006"
  6.     RFC822      = "02 Jan 06 15:04 MST"
  7.     RFC822Z     = "02 Jan 06 15:04 -0700" // RFC822 with numeric zone
  8.     RFC850      = "Monday, 02-Jan-06 15:04:05 MST"
  9.     RFC1123     = "Mon, 02 Jan 2006 15:04:05 MST"
  10.     RFC1123Z    = "Mon, 02 Jan 2006 15:04:05 -0700" // RFC1123 with numeric zone
  11.     RFC3339     = "2006-01-02T15:04:05Z07:00"
  12.     RFC3339Nano = "2006-01-02T15:04:05.999999999Z07:00"
  13.     Kitchen     = "3:04PM"
  14.     // Handy time stamps.
  15.     Stamp      = "Jan _2 15:04:05"
  16.     StampMilli = "Jan _2 15:04:05.000"
  17.     StampMicro = "Jan _2 15:04:05.000000"
  18.     StampNano  = "Jan _2 15:04:05.000000000"
  19. )

It can also be modified according to requirements.

regex(key)

Use regular pattern to segment logs and extract fields. It can also be regex(key, to).

parameter:

  • key: required, the field to be extracted regularly.
  • to: optional, the key to which all fields are placed after extraction. The default is empty, which means to extract the field to the root.

Extra fields:

  • pattern: required, regular expression.

Example

  1. - action: regex(body)
  2.   pattern: (?<ip>\S+) (?<id>\S+) (?<u>\S+) (?<time>\[.*?\]) (?<url>\".*?\") (?<status>\S+) (?<size>\S+)

input:

  1. {
  2.   "body": "10.244.0.1 - - [13/Dec/2021:12:40:48 +0000] 'GET / HTTP/1.1' 404 683",
  3. }

output:

  1. {
  2.   "ip":     "10.244.0.1",
  3.   "id":     "-",
  4.   "u":      "-",
  5.   "time":   "[13/Dec/2021:12:40:48 +0000]",
  6.   "url":    "GET / HTTP/1.1",
  7.   "status": "404",
  8.   "size":   "683",
  9. }

jsonDecode(key)

Deserialize json text. Can also be jsonDecode(key, to).

parameter:

  • key: required, the corresponding field key.
  • to: optional, the key to which all fields are placed after extraction. The default is empty, which means to extract the field to the root.

Example

  1. - action: jsonDecode(body)

input:

  1. {
  2.   "body": `{"log":"I0610 08:29:07.698664 Waiting for caches to sync", "stream":"stderr", "time":"2021-06-10T08:29:07.698731204Z"}`,
  3. }

output:

  1. {
  2.   "log": "I0610 08:29:07.698664 Waiting for caches to sync",
  3.   "stream": "stderr", 
  4.   "time": "2021-06-10T08:29:07.698731204Z"
  5. }

strconv(key, type)

Value type conversion.

parameter:

  • key: target field
  • type: the converted type, which can be bool, int, float

Example

  1. - action: strconv(code, int)

input:

  1. {
  2.   "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
  3.   "code": "200"
  4. }

output:

  1. {
  2.   "body": "2021-02-16T09:21:20.545525544Z DEBUG this is log body",
  3.   "code": 200
  4. }

print()

Print event. Generally used in the debugging phase.

return()

Control function. Interceptor returns when executing return(), and does not continue to execute the following action.

dropEvent()

Control function. Interceptor drops event when executing dropEvent(), which means that the piece of data will be lost and will not continue to be processed and consumed by subsequent interceptors or sinks.

Example

  1. interceptors:
  2.   - type: transformer
  3.     actions:
  4.       - action: regex(body)
  5.         pattern: ^(?P<time>[^ ^Z]+Z) (?P<level>[^ ]*) (?P<log>.*)$
  6.       - if: equal(level, DEBUG)
  7.         then:
  8.           - action: dropEvent()

Assuming that the log is: 2021-02-16T09:21:20.545525544Z DEBUG this is log body.If the level field is DEBUG, the log will be discarded directly.

condition

Conditional function.

operator

  • AND: Indicates the and result of two conditions

Example

  1. interceptors:
  2.   - type: transformer
  3.     actions:
  4.       - if: equal(level, DEBUG) AND equal(code, 200)
  5.         then:
  6.           - action: dropEvent()
  • OR：Indicates the or result of two conditions

Example

  1. interceptors:
  2.   - type: transformer
  3.     actions:
  4.       - if: equal(level, DEBUG) OR equal(level, INFO)
  5.         then:
  6.           - action: dropEvent()
  • NOT: Indicates the not result of the condition

Example

  1. interceptors:
  2.   - type: transformer
  3.     actions:
  4.       - if: NOT equal(level, DEBUG)
  5.         then:
  6.           - action: dropEvent()

equal(key, target)

Whether the value of field is equal to target.

contain(key, target)

Whether the value of field contains target.

exist(key)

Whether the field exists or is empty.

greater(key, target)

Whether the value of field is greater than targrt.

less(key, target)

Whether the value of field is less than targrt.

hasPrefix(key, target)

Whether the value of field contains the target prefix.

match(key, regex)

Whether the value of field matches regex.

oneOf(key, value1, value2…)

Whether the value of field is one of the value1 ​​value2…