import strict mode

Strict mode (strict_mode) is configured as a parameter in the import operation. This parameter affects the import behavior of certain values and the final imported data.

This document mainly explains how to set strict mode, and the impact of strict mode.

How to set

Strict mode is all False by default, i.e. off.

Different import methods set strict mode in different ways.

  1. BROKER LOAD

    1. LOAD LABEL example_db.label1
    2. (
    3.     DATA INFILE("bos://my_bucket/input/file.txt")
    4.     INTO TABLE `my_table`
    5.     COLUMNS TERMINATED BY ","
    6. )
    7. WITH BROKER bos
    8. (
    9.     "bos_endpoint" = "http://bj.bcebos.com",
    10.     "bos_accesskey" = "xxxxxxxxxxxxxxxxxxxxxxxxxxx",
    11.     "bos_secret_accesskey"="yyyyyyyyyyyyyyyyyyyyyyyy"
    12. )
    13. PROPERTIES
    14. (
    15.     "strict_mode" = "true"
    16. )

  2. STREAM LOAD

    1. curl --location-trusted -u user:passwd \
    2. -H "strict_mode: true" \
    3. -T 1.txt \
    4. http://host:port/api/example_db/my_table/_stream_load

  3. ROUTINE LOAD

    1. CREATE ROUTINE LOAD example_db.test_job ON my_table
    2. PROPERTIES
    3. (
    4.     "strict_mode" = "true"
    5. )
    6. FROM KAFKA
    7. (
    8.     "kafka_broker_list" = "broker1:9092,broker2:9092,broker3:9092",
    9.     "kafka_topic" = "my_topic"
    10. );

  4. INSERT

    Set via session variables:

    1. SET enable_insert_strict = true;
    2. INSERT INTO my_table ...;

The role of strict mode

Strict mode means strict filtering of column type conversions during import.

The strict filtering strategy is as follows:

For column type conversion, if strict mode is turned on, the wrong data will be filtered. The wrong data here refers to: the original data is not null, but the result is null after column type conversion.

The column type conversion mentioned here does not include the null value calculated by the function.

For an imported column type that contains range restrictions, if the original data can pass the type conversion normally, but cannot pass the range restrictions, strict mode will not affect it. For example: if the type is decimal(1,0) and the original data is 10, it belongs to the range that can be converted by type but is not within the scope of the column declaration. This kind of data strict has no effect on it.

  1. Take the column type as TinyInt as an example:

    Primitive data typePrimitive data exampleConverted value to TinyIntStrict modeResult
    NULL\NNULLON or OFFNULL
    Non-null value“abc” or 2000NULLOnIllegal value (filtered)
    non-null value“abc”NULLoffNULL
    non-null value11on or offimport correctly

    Description:

    1. Columns in the table allow to import null values
    2. After abc and 2000 are converted to TinyInt, they will become NULL due to type or precision issues. When strict mode is on, this data will be filtered. And if it is closed, null will be imported.

  2. Take the column type as Decimal(1,0) as an example

    Primitive Data TypesExamples of Primitive DataConverted to DecimalStrict ModeResult
    Null\NnullOn or OffNULL
    non-null valueaaaNULLonillegal value (filtered)
    non-null valueaaaNULLoffNULL
    non-null value1 or 101 or 10on or offimport correctly

    Description:

    1. Columns in the table allow to import null values
    2. After abc is converted to Decimal, it will become NULL due to type problem. When strict mode is on, this data will be filtered. And if it is closed, null will be imported.
    3. Although 10 is an out-of-range value, because its type conforms to the requirements of decimal, strict mode does not affect it. 10 will eventually be filtered in other import processing flows. But not filtered by strict mode.