JSON format data import

Doris supports importing data in JSON format. This document mainly describes the precautions when importing data in JSON format.

Supported import methods

Currently, only the following import methods support data import in Json format:

  • Import the local JSON format file through STREAM LOAD.
  • Subscribe and consume JSON format in Kafka via ROUTINE LOAD information.

Other ways of importing data in JSON format are not currently supported.

Supported Json formats

Currently only the following two Json formats are supported:

  1. Multiple rows of data represented by Array

    Json format with Array as root node. Each element in the Array represents a row of data to be imported, usually an Object. An example is as follows:

    1. [
    2.     { "id": 123, "city" : "beijing"},
    3.     { "id": 456, "city" : "shanghai"},
    4.     ...
    5. ]
    1. [
    2.     { "id": 123, "city" : { "name" : "beijing", "region" : "haidian"}},
    3.     { "id": 456, "city" : { "name" : "beijing", "region" : "chaoyang"}},
    4.     ...
    5. ]

    This method is typically used for Stream Load import methods to represent multiple rows of data in a batch of imported data.

    This method must be used with the setting strip_outer_array=true. Doris will expand the array when parsing, and then parse each Object in turn as a row of data.

  2. A single row of data represented by Object Json format with Object as root node. The entire Object represents a row of data to be imported. An example is as follows:

    1. { "id": 123, "city" : "beijing"}
    1. { "id": 123, "city" : { "name" : "beijing", "region" : "haidian" }}

    This method is usually used for the Routine Load import method, such as representing a message in Kafka, that is, a row of data.

  3. Multiple lines of Object data separated by a fixed delimiter

    A row of data represented by Object represents a row of data to be imported. The example is as follows:

    1. { "id": 123, "city" : "beijing"}
    2. { "id": 456, "city" : "shanghai"}
    3. ...

    This method is typically used for Stream Load import methods to represent multiple rows of data in a batch of imported data.

    This method must be used with the setting read_json_by_line=true, the special delimiter also needs to specify the line_delimiter parameter, the default is \n. When Doris parses, it will be separated according to the delimiter, and then parse each line of Object as a line of data.

streaming_load_json_max_mb parameters

Some data formats, such as JSON, cannot be split. Doris must read all the data into the memory before parsing can begin. Therefore, this value is used to limit the maximum amount of data that can be loaded in a single Stream load.

The default value is 100, The unit is MB, modify this parameter by referring to the BE configuration.

fuzzy_parse parameters

In STREAM LOAD fuzzy_parse parameter can be added to speed up JSON Data import efficiency.

This parameter is usually used to import the format of multi-line data represented by Array, so it is generally used with strip_outer_array=true.

This feature requires that each row of data in the Array has exactly the same order of fields. Doris will only parse according to the field order of the first row, and then access the subsequent data in the form of subscripts. This method can improve the import efficiency by 3-5X.

Json Path

Doris supports extracting data specified in Json through Json Path.

Note: Because for Array type data, Doris will expand the array first, and finally process it in a single line according to the Object format. So the examples later in this document are all explained with Json data in a single Object format.

  • do not specify Json Path

    If Json Path is not specified, Doris will use the column name in the table to find the element in Object by default. An example is as follows:

    The table contains two columns: id, city

    The Json data is as follows:

    1. { "id": 123, "city" : "beijing"}

    Then Doris will use id, city for matching, and get the final data 123 and beijing.

    If the Json data is as follows:

    1. { "id": 123, "name" : "beijing"}

    Then use id, city for matching, and get the final data 123 and null.

  • Specify Json Path

    Specify a set of Json Path in the form of a Json data. Each element in the array represents a column to extract. An example is as follows:

    1. ["$.id", "$.name"]
    1. ["$.id.sub_id", "$.name[0]", "$.city[0]"]

    Doris will use the specified Json Path for data matching and extraction.

  • matches non-primitive types

    The values that are finally matched in the preceding examples are all primitive types, such as integers, strings, and so on. Doris currently does not support composite types, such as Array, Map, etc. So when a non-basic type is matched, Doris will convert the type to a string in Json format and import it as a string type. An example is as follows:

    The Json data is:

    1. { "id": 123, "city" : { "name" : "beijing", "region" : "haidian" }}

    Json Path is ["$.city"]. The matched elements are:

    1. { "name" : "beijing", "region" : "haidian" }

    The element will be converted to a string for subsequent import operations:

    1. "{'name':'beijing','region':'haidian'}"

  • match failed

    When the match fails, null will be returned. An example is as follows:

    The Json data is:

    1. { "id": 123, "name" : "beijing"}

    Json Path is ["$.id", "$.info"]. The matched elements are 123 and null.

    Doris currently does not distinguish between null values represented in Json data and null values produced when a match fails. Suppose the Json data is:

    1. { "id": 123, "name" : null }

    The same result would be obtained with the following two Json Paths: 123 and null.

    1. ["$.id", "$.name"]
    1. ["$.id", "$.info"]

  • Exact match failed

    In order to prevent misoperation caused by some parameter setting errors. When Doris tries to match a row of data, if all columns fail to match, it considers this to be an error row. Suppose the Json data is:

    1. { "id": 123, "city" : "beijing" }

    If the Json Path is written incorrectly as (or if the Json Path is not specified, the columns in the table do not contain id and city):

    1. ["$.ad", "$.infa"]

    would cause the exact match to fail, and the line would be marked as an error line instead of yielding null, null.

Json Path and Columns

Json Path is used to specify how to extract data in JSON format, while Columns specifies the mapping and conversion relationship of columns. Both can be used together.

In other words, it is equivalent to rearranging the columns of a Json format data according to the column order specified in Json Path through Json Path. After that, you can map the rearranged source data to the columns of the table through Columns. An example is as follows:

Data content:

  1. {"k1": 1, "k2": 2}

Table Structure:

  1. k2 int, k1 int

Import statement 1 (take Stream Load as an example):

  1. curl -v --location-trusted -u root: -H "format: json" -H "jsonpaths: [\"$.k2\", \"$.k1\"]" -T example.json http:/ /127.0.0.1:8030/api/db1/tbl1/_stream_load

In import statement 1, only Json Path is specified, and Columns is not specified. The role of Json Path is to extract the Json data in the order of the fields in the Json Path, and then write it in the order of the table structure. The final imported data results are as follows:

  1. +------+------+
  2. | k1 | k2 |
  3. +------+------+
  4. | 2 | 1 |
  5. +------+------+

You can see that the actual k1 column imports the value of the “k2” column in the Json data. This is because the field name in Json is not equivalent to the field name in the table structure. We need to explicitly specify the mapping between the two.

Import statement 2:

  1. curl -v --location-trusted -u root: -H "format: json" -H "jsonpaths: [\"$.k2\", \"$.k1\"]" -H "columns: k2, k1 " -T example.json http://127.0.0.1:8030/api/db1/tbl1/_stream_load

Compared with the import statement 1, the Columns field is added here to describe the mapping relationship of the columns, in the order of k2, k1. That is, after extracting in the order of fields in Json Path, specify the value of column k2 in the table for the first column, and the value of column k1 in the table for the second column. The final imported data results are as follows:

  1. +------+------+
  2. | k1 | k2 |
  3. +------+------+
  4. | 1 | 2 |
  5. +------+------+

Of course, as with other imports, column transformations can be performed in Columns. An example is as follows:

  1. curl -v --location-trusted -u root: -H "format: json" -H "jsonpaths: [\"$.k2\", \"$.k1\"]" -H "columns: k2, tmp_k1 , k1 = tmp_k1 * 100" -T example.json http://127.0.0.1:8030/api/db1/tbl1/_stream_load

The above example will import the value of k1 multiplied by 100. The final imported data results are as follows:

  1. +------+------+
  2. | k1 | k2 |
  3. +------+------+
  4. | 100 | 2 |
  5. +------+------+

Json root

Doris supports extracting data specified in Json through Json root.

Note: Because for Array type data, Doris will expand the array first, and finally process it in a single line according to the Object format. So the examples later in this document are all explained with Json data in a single Object format.

  • do not specify Json root

    If Json root is not specified, Doris will use the column name in the table to find the element in Object by default. An example is as follows:

    The table contains two columns: id, city

    The Json data is as follows:

    1. { "id": 123, "name" : { "id" : "321", "city" : "shanghai" }}

    Then use id, city for matching, and get the final data 123 and null

  • Specify Json root

    When the import data format is json, you can specify the root node of the Json data through json_root. Doris will extract the elements of the root node through json_root for parsing. Default is empty.

    Specify Json root -H "json_root: $.name". The matched elements are:

    1. { "id" : "321", "city" : "shanghai" }

    The element will be treated as new json for subsequent import operations,and get the final data 321 and shanghai

NULL and Default values

Example data is as follows:

  1. [
  2.     {"k1": 1, "k2": "a"},
  3.     {"k1": 2},
  4.     {"k1": 3, "k2": "c"},
  5. ]

The table structure is: k1 int null, k2 varchar(32) null default "x"

The import statement is as follows:

  1. curl -v --location-trusted -u root: -H "format: json" -H "strip_outer_array: true" -T example.json http://127.0.0.1:8030/api/db1/tbl1/_stream_load

The import results that users may expect are as follows, that is, for missing columns, fill in the default values.

  1. +------+------+
  2. | k1 | k2 |
  3. +------+------+
  4. | 1 | a |
  5. +------+------+
  6. | 2 | x |
  7. +------+------+
  8. |3|c|
  9. +------+------+

But the actual import result is as follows, that is, for the missing column, NULL is added.

  1. +------+------+
  2. | k1 | k2 |
  3. +------+------+
  4. | 1 | a |
  5. +------+------+
  6. | 2 | NULL |
  7. +------+------+
  8. |3|c|
  9. +------+------+

This is because Doris doesn’t know “the missing column is column k2 in the table” from the information in the import statement. If you want to import the above data according to the expected result, the import statement is as follows:

  1. curl -v --location-trusted -u root: -H "format: json" -H "strip_outer_array: true" -H "jsonpaths: [\"$.k1\", \"$.k2\"]" - H "columns: k1, tmp_k2, k2 = ifnull(tmp_k2, 'x')" -T example.json http://127.0.0.1:8030/api/db1/tbl1/_stream_load

Application example

Stream Load

Because of the inseparability of the Json format, when using Stream Load to import a Json format file, the file content will be fully loaded into the memory before processing begins. Therefore, if the file is too large, it may take up more memory.

Suppose the table structure is:

  1. id INT NOT NULL,
  2. city VARHCAR NULL,
  3. code INT NULL

  1. Import a single row of data 1

    1. {"id": 100, "city": "beijing", "code" : 1}

    • do not specify Json Path

      1. curl --location-trusted -u user:passwd -H "format: json" -T data.json http://localhost:8030/api/db1/tbl1/_stream_load

      Import result:

      1. 100 beijing 1

    • Specify Json Path

      1. curl --location-trusted -u user:passwd -H "format: json" -H "jsonpaths: [\"$.id\",\"$.city\",\"$.code\"]" - T data.json http://localhost:8030/api/db1/tbl1/_stream_load

      Import result:

      1. 100 beijing 1

  2. Import a single row of data 2

    1. {"id": 100, "content": {"city": "beijing", "code": 1}}

    • Specify Json Path

      1. curl --location-trusted -u user:passwd -H "format: json" -H "jsonpaths: [\"$.id\",\"$.content.city\",\"$.content.code\ "]" -T data.json http://localhost:8030/api/db1/tbl1/_stream_load

      Import result:

      1. 100 beijing 1

  3. Import multiple rows of data as Array

    1. [
    2.     {"id": 100, "city": "beijing", "code" : 1},
    3.     {"id": 101, "city": "shanghai"},
    4.     {"id": 102, "city": "tianjin", "code" : 3},
    5.     {"id": 103, "city": "chongqing", "code" : 4},
    6.     {"id": 104, "city": ["zhejiang", "guangzhou"], "code" : 5},
    7.     {
    8.         "id": 105,
    9.         "city": {
    10.             "order1": ["guangzhou"]
    11.         },
    12.         "code" : 6
    13.     }
    14. ]

    • Specify Json Path

      1. curl --location-trusted -u user:passwd -H "format: json" -H "jsonpaths: [\"$.id\",\"$.city\",\"$.code\"]" - H "strip_outer_array: true" -T data.json http://localhost:8030/api/db1/tbl1/_stream_load

      Import result:

      1. 100 beijing 1
      2. 101 shanghai NULL
      3. 102 tianjin 3
      4. 103 chongqing 4
      5. 104 ["zhejiang","guangzhou"] 5
      6. 105 {"order1":["guangzhou"]} 6

  4. Import multi-line data as multi-line Object

    1. {"id": 100, "city": "beijing", "code" : 1}
    2. {"id": 101, "city": "shanghai"}
    3. {"id": 102, "city": "tianjin", "code" : 3}
    4. {"id": 103, "city": "chongqing", "code" : 4}

    StreamLoad import：

  1. curl --location-trusted -u user:passwd -H "format: json" -H "read_json_by_line: true" -T data.json http://localhost:8030/api/db1/tbl1/_stream_load
  1.    Import result:
  3. 100     beijing                     1
  4. 101     shanghai                    NULL
  5. 102     tianjin                     3
  6. 103     chongqing                   4
  1. Transform the imported data

The data is still the multi-line data in Example 3, and now it is necessary to add 1 to the code column in the imported data before importing.

  1. curl --location-trusted -u user:passwd -H "format: json" -H "jsonpaths: [\"$.id\",\"$.city\",\"$.code\"]" - H "strip_outer_array: true" -H "columns: id, city, tmpc, code=tmpc+1" -T data.json http://localhost:8030/api/db1/tbl1/_stream_load

Import result:

  1. 100 beijing 2
  2. 101 shanghai NULL
  3. 102 tianjin 4
  4. 103 chongqing 5
  5. 104 ["zhejiang","guangzhou"] 6
  6. 105 {"order1":["guangzhou"]} 7

Routine Load

The processing principle of Routine Load for Json data is the same as that of Stream Load. It is not repeated here.

For Kafka data sources, the content in each Massage is treated as a complete Json data. If there are multiple rows of data represented in Array format in a Massage, multiple rows will be imported, and the offset of Kafka will only increase by 1. If an Array format Json represents multiple lines of data, but the Json parsing fails due to the wrong Json format, the error line will only increase by 1 (because the parsing fails, in fact, Doris cannot determine how many lines of data are contained in it, and can only error by one line data record)