normalize

(The interceptor will no longer be maintained in the future, and it is recommended to replace it with a transformer)

Used for log segmentation.
Source interceptor. Can be specified to be used only by certain sources.

processors

The configured processors will be executed in order.

Tips

Loggie supports using a.b to refer to nested fields. For example:

{
  "fields": {
    "hello": "world"
  }
}

The following processor can use fields.hello to specific hello: world inside fields.

addMeta

By default, Loggie does not add any internal system information to the raw data. By addMeta, the built-in fields of the system can be added and sent to the downstream.

Note

Please note that configuring addMeta in a pipeline will only affect all data sent by the pipeline. If you need the interceptor to take effect globally, please configure normalize.addMeta in defaults.

loggie:
  defaults:
    interceptors:
    - type: normalize
      name: global
      processors:
       - addMeta: ~

regex

Regularly extract the specified field.

Example

interceptors:
- type: normalize
  processors:
  - regex:
      pattern: '(?<ip>\S+) (?<id>\S+) (?<u>\S+) (?<time>\[.*?\]) (?<url>\".*?\") (?<status>\S+) (?<size>\S+)'

Using the above regular expression, the log of the following example can be converted from:

10.244.0.1 - - [13/Dec/2021:12:40:48 +0000] "GET / HTTP/1.1" 404 683

to:

"ip": "10.244.0.1",
"id": "-",
"u": "-",
"time": "[13/Dec/2021:12:40:48 +0000]",
"url": "\"GET / HTTP/1.1\"",
"status": "404",
"size": "683"

When configuring the specific configuration, it is recommended to use some regular debugging tools (https://regex101.com/) to verify whether it works.

jsonDecode

Parse the specified field json.

Example

interceptors:
- type: normalize
  processors:
  - jsonDecode: ~

split

Extract the specified field with a delimiter.

Example

base

interceptors:
- type: normalize
  processors:
  - split:
      separator: '|'
      keys: ["time", "order", "service", "price"]

Using the above split configuration can convert the log from:

2021-08-08|U12345|storeCenter|13.14

to:

"time": "2021-08-08"
"order": "U12345"
"service": "storeCenter"
"price": 13.14

max

interceptors:
- type: normalize
  processors:
  - split:
      separator: ' '
      max: 2
      keys: ["time", "content"]

By adding max, you can control the fields that are split at most. For example the following log:

2021-08-08 U12345 storeCenter 13.14

It can be converted by the above configuration as:

"time": "2021-08-08"
"content": "U12345 storeCenter 13.14"

drop

Discards the specified field.

Example

interceptors:
- type: normalize
  processors:
  - drop:
      targets: ["id", "body"]

rename

Rename the specified field.

Example

interceptors:
- type: normalize
  processors:
  - rename:
      convert:
      - from: "hello"
        to: "world"

add

Add fields.

Example

interceptors:
- type: normalize
  processors:
  - add:
      fields:
        hello: world

convert

Field type conversion.

Example

interceptors:
- type: normalize
  processors:
  - convert:
      convert:
      - from: count
        to: float

copy

Field copy.

Example

interceptors:
- type: normalize
  processors:
  - copy:
      convert:
      - from: hello
        to: world

underRoot

Put all the fields in the key:value outermost layer of the event.

Example

interceptors:
- type: normalize
  processors:
  - underRoot:
      keys: ["fields"]

timestamp

Convert time format.

Example

interceptors:
- type: normalize
  processors:
  - timestamp:
      convert:
      - from: logtime
        fromLayout: "2006-01-02T15:04:05Z07:00"
        toLayout: "unix"

For time in golang form, please refer to:

const (
    Layout      = "01/02 03:04:05PM '06 -0700" // The reference time, in numerical order.
    ANSIC       = "Mon Jan _2 15:04:05 2006"
    UnixDate    = "Mon Jan _2 15:04:05 MST 2006"
    RubyDate    = "Mon Jan 02 15:04:05 -0700 2006"
    RFC822      = "02 Jan 06 15:04 MST"
    RFC822Z     = "02 Jan 06 15:04 -0700" // RFC822 with numeric zone
    RFC850      = "Monday, 02-Jan-06 15:04:05 MST"
    RFC1123     = "Mon, 02 Jan 2006 15:04:05 MST"
    RFC1123Z    = "Mon, 02 Jan 2006 15:04:05 -0700" // RFC1123 with numeric zone
    RFC3339     = "2006-01-02T15:04:05Z07:00"
    RFC3339Nano = "2006-01-02T15:04:05.999999999Z07:00"
    Kitchen     = "3:04PM"
    // Handy time stamps.
    Stamp      = "Jan _2 15:04:05"
    StampMilli = "Jan _2 15:04:05.000"
    StampMicro = "Jan _2 15:04:05.000000"
    StampNano  = "Jan _2 15:04:05.000000000"
)

It can also be modified according to requirements.

fmt

Reformat field contents. Combination and formatting according to other field contents are supported.

Example

interceptors:
- type: normalize
  processors:
  - fmt:
      fields:
        d: new-${a.b}-${c}