list_to_map

The list_to_map processor converts a list of objects from an event, where each object contains a key field, into a map of target keys.

Configuration

The following table describes the configuration options used to generate target keys for the mappings.

OptionRequiredTypeDescription
keyYesStringThe key of the fields to be extracted as keys in the generated mappings.
sourceYesStringThe list of objects with key fields to be converted into keys for the generated map.
targetNoStringThe target for the generated map. When not specified, the generated map will be placed in the root node.
value_keyNoStringWhen specified, values given a value_key in objects contained in the source list will be extracted and converted into the value specified by this option based on the generated map. When not specified, objects contained in the source list retain their original value when mapped.
flattenNoBooleanWhen true, values in the generated map output flatten into single items based on the flattened_element. Otherwise, objects mapped to values from the generated map appear as lists.
flattened_elementConditionallyStringThe element to keep, either first or last, when flatten is set to true.

Usage

The following example shows how to test the usage of the list_to_map processor before using the processor on your own source.

Create a source file named logs_json.log. Because the file source reads each line in the .log file as an event, the object list appears as one line even though it contains multiple objects:

  1. {"mylist":[{"name":"a","value":"val-a"},{"name":"b","value":"val-b1"},{"name":"b",  "value":"val-b2"},{"name":"c","value":"val-c"}]}

copy

Next, create a pipeline.yaml file that uses the logs_json.log file as the source by pointing to the .log file’s correct path:

  1. pipeline:
  2.   source:
  3.     file:
  4.       path: "/full/path/to/logs_json.log"
  5.       record_type: "event"
  6.       format: "json"
  7.   processor:
  8.     - list_to_map:
  9.         key: "name"
  10.         source: "mylist"
  11.         value_key: "value"
  12.         flatten: true
  13.   sink:
  14.     - stdout:

copy

Run the pipeline. If successful, the processor returns the generated map with objects mapped according to their value_key. Similar to the original source, which contains one line and therefore one event, the processor returns the following JSON as one line. For readability, the following example and all subsequent JSON examples have been adjusted to span multiple lines:

  1. {
  2.   "mylist": [
  3.     {
  4.       "name": "a",
  5.       "value": "val-a"
  6.     },
  7.     {
  8.       "name": "b",
  9.       "value": "val-b1"
  10.     },
  11.     {
  12.       "name": "b",
  13.       "value": "val-b2"
  14.     },
  15.     {
  16.       "name": "c",
  17.       "value": "val-c"
  18.     }
  19.   ],
  20.   "a": "val-a",
  21.   "b": "val-b1",
  22.   "c": "val-c"
  23. }

Example: Maps set to target

The following example pipeline.yaml file shows the list_to_map processor when set to a specified target, mymap:

  1. pipeline:
  2.   source:
  3.     file:
  4.       path: "/full/path/to/logs_json.log"
  5.       record_type: "event"
  6.       format: "json"
  7.   processor:
  8.     - list_to_map:
  9.         key: "name"
  10.         source: "mylist"
  11.         target: "mymap"
  12.         value_key: "value"
  13.         flatten: true
  14.   sink:
  15.     - stdout:

copy

The generated map appears under the target key:

  1. {
  2.   "mylist": [
  3.     {
  4.       "name": "a",
  5.       "value": "val-a"
  6.     },
  7.     {
  8.       "name": "b",
  9.       "value": "val-b1"
  10.     },
  11.     {
  12.       "name": "b",
  13.       "value": "val-b2"
  14.     },
  15.     {
  16.       "name": "c",
  17.       "value": "val-c"
  18.     }
  19.   ],
  20.   "mymap": {
  21.     "a": "val-a",
  22.     "b": "val-b1",
  23.     "c": "val-c"
  24.   }
  25. }

Example: No value_key specified

The follow example pipeline.yaml file shows the list_to_map processor with no value_key specified. Because key is set to name, the processor extracts the object names to use as keys in the map.

  1. pipeline:
  2.   source:
  3.     file:
  4.       path: "/full/path/to/logs_json.log"
  5.       record_type: "event"
  6.       format: "json"
  7.   processor:
  8.     - list_to_map:
  9.         key: "name"
  10.         source: "mylist"
  11.         flatten: true
  12.   sink:
  13.     - stdout:

copy

The values from the generated map appear as original objects from the .log source, as shown in the following example response:

  1. {
  2.   "mylist": [
  3.     {
  4.       "name": "a",
  5.       "value": "val-a"
  6.     },
  7.     {
  8.       "name": "b",
  9.       "value": "val-b1"
  10.     },
  11.     {
  12.       "name": "b",
  13.       "value": "val-b2"
  14.     },
  15.     {
  16.       "name": "c",
  17.       "value": "val-c"
  18.     }
  19.   ],
  20.   "a": {
  21.     "name": "a",
  22.     "value": "val-a"
  23.   },
  24.   "b": {
  25.     "name": "b",
  26.     "value": "val-b1"
  27.   },
  28.   "c": {
  29.     "name": "c",
  30.     "value": "val-c"
  31.   }
  32. }

Example: flattened_element set to last

The following example pipeline.yaml file sets the flattened_element to last, therefore flattening the processor output based on each value’s last element:

  1. pipeline:
  2.   source:
  3.     file:
  4.       path: "/full/path/to/logs_json.log"
  5.       record_type: "event"
  6.       format: "json"
  7.   processor:
  8.     - list_to_map:
  9.         key: "name"
  10.         source: "mylist"
  11.         target: "mymap"
  12.         value_key: "value"
  13.         flatten: true
  14.         flattened_element: "last"
  15.   sink:
  16.     - stdout:

copy

The processor maps object b to value val-b2 because val-b2 is the last element in object b, as shown in the following output:

  1. {
  2.   "mylist": [
  3.     {
  4.       "name": "a",
  5.       "value": "val-a"
  6.     },
  7.     {
  8.       "name": "b",
  9.       "value": "val-b1"
  10.     },
  11.     {
  12.       "name": "b",
  13.       "value": "val-b2"
  14.     },
  15.     {
  16.       "name": "c",
  17.       "value": "val-c"
  18.     }
  19.   ],
  20.   "a": "val-a",
  21.   "b": "val-b2",
  22.   "c": "val-c"
  23. }

Example: flatten set to false

The following example pipeline.yaml file sets flatten to false, causing the processor to output values from the generated map as a list:

  1. pipeline:
  2.   source:
  3.     file:
  4.       path: "/full/path/to/logs_json.log"
  5.       record_type: "event"
  6.       format: "json"
  7.   processor:
  8.     - list_to_map:
  9.         key: "name"
  10.         source: "mylist"
  11.         target: "mymap"
  12.         value_key: "value"
  13.         flatten: false
  14.   sink:
  15.     - stdout:

copy

Some objects in the response may have more than one element in their values, as shown in the following response:

  1. {
  2.   "mylist": [
  3.     {
  4.       "name": "a",
  5.       "value": "val-a"
  6.     },
  7.     {
  8.       "name": "b",
  9.       "value": "val-b1"
  10.     },
  11.     {
  12.       "name": "b",
  13.       "value": "val-b2"
  14.     },
  15.     {
  16.       "name": "c",
  17.       "value": "val-c"
  18.     }
  19.   ],
  20.   "a": [
  21.     "val-a"
  22.   ],
  23.   "b": [
  24.     "val-b1",
  25.     "val-b2"
  26.   ],
  27.   "c": [
  28.     "val-c"
  29.   ]
  30. }

list_to_map - 图1