Data Types

Due to historical reasons, before Flink 1.9, Flink’s Table & SQL API data types weretightly coupled to Flink’s TypeInformation. TypeInformation is used in the DataStreamand DataSet API and is sufficient to describe all information needed to serialize anddeserialize JVM-based objects in a distributed setting.

However, TypeInformation was not designed to represent logical types independent ofan actual JVM class. In the past, it was difficult to map SQL standard types to thisabstraction. Furthermore, some types were not SQL-compliant and introduced without abigger picture in mind.

Starting with Flink 1.9, the Table & SQL API will receive a new type system that serves as a long-termsolution for API stability and standard compliance.

Reworking the type system is a major effort that touches almost all user-facing interfaces. Therefore, itsintroduction spans multiple releases, and the community aims to finish this effort by Flink 1.10.

Due to the simultaneous addition of a new planner for table programs (see FLINK-11439),not every combination of planner and data type is supported. Furthermore, planners might not support everydata type with the desired precision or parameter.

Attention Please see the planner compatibility table and limitationssection before using a data type.

• Data Type
  • Data Types in the Table API
• Planner Compatibility
  • Old Planner
  • New Blink Planner
• Limitations
• List of Data Types
  • Character Strings
  • Binary Strings
  • Exact Numerics
  • Approximate Numerics
  • Date and Time
  • Constructured Data Types
  • Other Data Types

Data Type

A data type describes the logical type of a value in the table ecosystem. It can be used to declare input and/oroutput types of operations.

Flink’s data types are similar to the SQL standard’s data type terminology but also contain informationabout the nullability of a value for efficient handling of scalar expressions.

Examples of data types are:

• INT
• INT NOT NULL
• INTERVAL DAY TO SECOND(3)
• ROW<myField ARRAY<BOOLEAN>, myOtherField TIMESTAMP(3)>

A list of all pre-defined data types can be found below.

Data Types in the Table API

Users of the JVM-based API work with instances of org.apache.flink.table.types.DataType within the Table API or whendefining connectors, catalogs, or user-defined functions.

A DataType instance has two responsibilities:

• Declaration of a logical type which does not imply a concrete physical representation for transmissionor storage but defines the boundaries between JVM-based languages and the table ecosystem.
• Optional:Giving hints about the physical representation of data to the planner which is useful at the edges to other APIs .

For JVM-based languages, all pre-defined data types are available in org.apache.flink.table.api.DataTypes.

It is recommended to add a star import to your table programs for having a fluent API:

import static org.apache.flink.table.api.DataTypes.*;
DataType t = INTERVAL(DAY(), SECOND(3));

import org.apache.flink.table.api.DataTypes._
val t: DataType = INTERVAL(DAY(), SECOND(3));

Physical Hints

Physical hints are required at the edges of the table ecosystem where the SQL-based type system ends andprogramming-specific data types are required. Hints indicate the data format that an implementationexpects.

For example, a data source could express that it produces values for logical TIMESTAMPs using a java.sql.Timestamp classinstead of using java.time.LocalDateTime which would be the default. With this information, the runtime is able to convertthe produced class into its internal data format. In return, a data sink can declare the data format it consumes from the runtime.

Here are some examples of how to declare a bridging conversion class:

// tell the runtime to not produce or consume java.time.LocalDateTime instances
// but java.sql.Timestamp
DataType t = DataTypes.TIMESTAMP(3).bridgedTo(java.sql.Timestamp.class);
// tell the runtime to not produce or consume boxed integer arrays
// but primitive int arrays
DataType t = DataTypes.ARRAY(DataTypes.INT().notNull()).bridgedTo(int[].class);

// tell the runtime to not produce or consume java.time.LocalDateTime instances
// but java.sql.Timestamp
val t: DataType = DataTypes.TIMESTAMP(3).bridgedTo(classOf[java.sql.Timestamp]);
// tell the runtime to not produce or consume boxed integer arrays
// but primitive int arrays
val t: DataType = DataTypes.ARRAY(DataTypes.INT().notNull()).bridgedTo(classOf[Array[Int]]);

Attention Please note that physical hints are usually only required if theAPI is extended. Users of predefined sources/sinks/functions do not need to define such hints. Hints withina table program (e.g. field.cast(TIMESTAMP(3).bridgedTo(Timestamp.class))) are ignored.

Planner Compatibility

As mentioned in the introduction, reworking the type system will span multiple releases, and the support of each datatype depends on the used planner. This section aims to summarize the most significant differences.

Old Planner

Flink’s old planner, introduced before Flink 1.9, primarily supports type information. It has only limitedsupport for data types. It is possible to declare data types that can be translated into type information such that theold planner understands them.

The following table summarizes the difference between data type and type information. Most simple types, as well as therow type remain the same. Time types, array types, and the decimal type need special attention. Other hints as the onesmentioned are not allowed.

For the Type Information column the table omits the prefix org.apache.flink.table.api.Types.

For the Data Type Representation column the table omits the prefix org.apache.flink.table.api.DataTypes.

Attention If there is a problem with the new type system. Userscan fallback to type information defined in org.apache.flink.table.api.Types at any time.

New Blink Planner

The new Blink planner supports all of types of the old planner. This includes in particularthe listed Java expression strings and type information.

The following data types are supported:

Limitations

Java Expression String: Java expression strings in the Table API such as table.select("field.cast(STRING)")have not been updated to the new type system yet. Use the string representations declared inthe old planner section.

Connector Descriptors and SQL Client: Descriptor string representations have not been updated to the newtype system yet. Use the string representation declared in the Connect to External Systems section

User-defined Functions: User-defined functions cannot declare a data type yet.

List of Data Types

This section lists all pre-defined data types. For the JVM-based Table API those types are also available in org.apache.flink.table.api.DataTypes.

Character Strings

CHAR

Data type of a fixed-length character string.

Declaration

CHAR
CHAR(n)

DataTypes.CHAR(n)

The type can be declared using CHAR(n) where n is the number of code points. n must have a value between 1and 2,147,483,647 (both inclusive). If no length is specified, n is equal to 1.

Bridging to JVM Types

VARCHAR / STRING

Data type of a variable-length character string.

Declaration

VARCHAR
VARCHAR(n)
STRING

DataTypes.VARCHAR(n)
DataTypes.STRING()

The type can be declared using VARCHAR(n) where n is the maximum number of code points. n must have a valuebetween 1 and 2,147,483,647 (both inclusive). If no length is specified, n is equal to 1.

STRING is a synonym for VARCHAR(2147483647).

Bridging to JVM Types

Binary Strings

BINARY

Data type of a fixed-length binary string (=a sequence of bytes).

Declaration

BINARY
BINARY(n)

DataTypes.BINARY(n)

The type can be declared using BINARY(n) where n is the number of bytes. n must have a valuebetween 1 and 2,147,483,647 (both inclusive). If no length is specified, n is equal to 1.

Bridging to JVM Types

VARBINARY / BYTES

Data type of a variable-length binary string (=a sequence of bytes).

Declaration

VARBINARY
VARBINARY(n)
BYTES

DataTypes.VARBINARY(n)
DataTypes.BYTES()

The type can be declared using VARBINARY(n) where n is the maximum number of bytes. n musthave a value between 1 and 2,147,483,647 (both inclusive). If no length is specified, n isequal to 1.

BYTES is a synonym for VARBINARY(2147483647).

Bridging to JVM Types

Exact Numerics

DECIMAL

Data type of a decimal number with fixed precision and scale.

Declaration

DECIMAL
DECIMAL(p)
DECIMAL(p, s)
DEC
DEC(p)
DEC(p, s)
NUMERIC
NUMERIC(p)
NUMERIC(p, s)

DataTypes.DECIMAL(p, s)

The type can be declared using DECIMAL(p, s) where p is the number of digits in anumber (precision) and s is the number of digits to the right of the decimal pointin a number (scale). p must have a value between 1 and 38 (both inclusive). smust have a value between 0 and p (both inclusive). The default value for p is 10.The default value for s is 0.

NUMERIC(p, s) and DEC(p, s) are synonyms for this type.

Bridging to JVM Types

TINYINT

Data type of a 1-byte signed integer with values from -128 to 127.

Declaration

TINYINT

DataTypes.TINYINT()

Bridging to JVM Types

SMALLINT

Data type of a 2-byte signed integer with values from -32,768 to 32,767.

Declaration

SMALLINT

DataTypes.SMALLINT()

Bridging to JVM Types

INT

Data type of a 4-byte signed integer with values from -2,147,483,648 to 2,147,483,647.

Declaration

INT
INTEGER

DataTypes.INT()

INTEGER is a synonym for this type.

Bridging to JVM Types

BIGINT

Data type of an 8-byte signed integer with values from -9,223,372,036,854,775,808 to9,223,372,036,854,775,807.

Declaration

BIGINT

DataTypes.BIGINT()

Bridging to JVM Types

Approximate Numerics

FLOAT

Data type of a 4-byte single precision floating point number.

Compared to the SQL standard, the type does not take parameters.

Declaration

FLOAT

DataTypes.FLOAT()

Bridging to JVM Types

DOUBLE

Data type of an 8-byte double precision floating point number.

Declaration

DOUBLE
DOUBLE PRECISION

DataTypes.DOUBLE()

DOUBLE PRECISION is a synonym for this type.

Bridging to JVM Types

Date and Time

DATE

Data type of a date consisting of year-month-day with values ranging from 0000-01-01to 9999-12-31.

Compared to the SQL standard, the range starts at year 0000.

Declaration

DATE

DataTypes.DATE()

Bridging to JVM Types

TIME

Data type of a time without time zone consisting of hour:minute:second[.fractional] withup to nanosecond precision and values ranging from 00:00:00.000000000 to23:59:59.999999999.

Compared to the SQL standard, leap seconds (23:59:60 and 23:59:61) are not supported asthe semantics are closer to java.time.LocalTime. A time with time zone is not provided.

Declaration

TIME
TIME(p)

DataTypes.TIME(p)

The type can be declared using TIME(p) where p is the number of digits of fractionalseconds (precision). p must have a value between 0 and 9 (both inclusive). If noprecision is specified, p is equal to 0.

Bridging to JVM Types

TIMESTAMP

Data type of a timestamp without time zone consisting of year-month-day hour:minute:second[.fractional]with up to nanosecond precision and values ranging from 0000-01-01 00:00:00.000000000 to9999-12-31 23:59:59.999999999.

Compared to the SQL standard, leap seconds (23:59:60 and 23:59:61) are not supported asthe semantics are closer to java.time.LocalDateTime.

A conversion from and to BIGINT (a JVM long type) is not supported as this would imply a timezone. However, this type is time zone free. For more java.time.Instant-like semantics useTIMESTAMP WITH LOCAL TIME ZONE.

Declaration

TIMESTAMP
TIMESTAMP(p)
TIMESTAMP WITHOUT TIME ZONE
TIMESTAMP(p) WITHOUT TIME ZONE

DataTypes.TIMESTAMP(p)

The type can be declared using TIMESTAMP(p) where p is the number of digits of fractionalseconds (precision). p must have a value between 0 and 9 (both inclusive). If no precisionis specified, p is equal to 6.

TIMESTAMP(p) WITHOUT TIME ZONE is a synonym for this type.

Bridging to JVM Types

TIMESTAMP WITH TIME ZONE

Data type of a timestamp with time zone consisting of year-month-day hour:minute:second[.fractional] zonewith up to nanosecond precision and values ranging from 0000-01-01 00:00:00.000000000 +14:59 to9999-12-31 23:59:59.999999999 -14:59.

Compared to the SQL standard, leap seconds (23:59:60 and 23:59:61) are not supported as the semanticsare closer to java.time.OffsetDateTime.

Compared to TIMESTAMP WITH LOCAL TIME ZONE, the time zone offset information is physicallystored in every datum. It is used individually for every computation, visualization, or communicationto external systems.

Declaration

TIMESTAMP WITH TIME ZONE
TIMESTAMP(p) WITH TIME ZONE

DataTypes.TIMESTAMP_WITH_TIME_ZONE(p)

The type can be declared using TIMESTAMP(p) WITH TIME ZONE where p is the number of digits offractional seconds (precision). p must have a value between 0 and 9 (both inclusive). If noprecision is specified, p is equal to 6.

Bridging to JVM Types

TIMESTAMP WITH LOCAL TIME ZONE

Data type of a timestamp with local time zone consisting of year-month-day hour:minute:second[.fractional] zonewith up to nanosecond precision and values ranging from 0000-01-01 00:00:00.000000000 +14:59 to9999-12-31 23:59:59.999999999 -14:59.

Leap seconds (23:59:60 and 23:59:61) are not supported as the semantics are closer to java.time.OffsetDateTime.

Compared to TIMESTAMP WITH TIME ZONE, the time zone offset information is not stored physicallyin every datum. Instead, the type assumes java.time.Instant semantics in UTC time zone atthe edges of the table ecosystem. Every datum is interpreted in the local time zone configured inthe current session for computation and visualization.

This type fills the gap between time zone free and time zone mandatory timestamp types by allowingthe interpretation of UTC timestamps according to the configured session time zone.

Declaration

TIMESTAMP WITH LOCAL TIME ZONE
TIMESTAMP(p) WITH LOCAL TIME ZONE

DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE(p)

The type can be declared using TIMESTAMP(p) WITH LOCAL TIME ZONE where p is the numberof digits of fractional seconds (precision). p must have a value between 0 and 9(both inclusive). If no precision is specified, p is equal to 6.

Bridging to JVM Types

INTERVAL YEAR TO MONTH

Data type for a group of year-month interval types.

The type must be parameterized to one of the following resolutions:

• interval of years,
• interval of years to months,
• or interval of months.

An interval of year-month consists of +years-months with values ranging from -9999-11 to+9999-11.

The value representation is the same for all types of resolutions. For example, an intervalof months of 50 is always represented in an interval-of-years-to-months format (with defaultyear precision): +04-02.

Declaration

INTERVAL YEAR
INTERVAL YEAR(p)
INTERVAL YEAR(p) TO MONTH
INTERVAL MONTH

DataTypes.INTERVAL(DataTypes.YEAR())
DataTypes.INTERVAL(DataTypes.YEAR(p))
DataTypes.INTERVAL(DataTypes.YEAR(p), DataTypes.MONTH())
DataTypes.INTERVAL(DataTypes.MONTH())

The type can be declared using the above combinations where p is the number of digits of years(year precision). p must have a value between 1 and 4 (both inclusive). If no year precisionis specified, p is equal to 2.

Bridging to JVM Types

INTERVAL DAY TO MONTH

Data type for a group of day-time interval types.

The type must be parameterized to one of the following resolutions with up to nanosecond precision:

• interval of days,
• interval of days to hours,
• interval of days to minutes,
• interval of days to seconds,
• interval of hours,
• interval of hours to minutes,
• interval of hours to seconds,
• interval of minutes,
• interval of minutes to seconds,
• or interval of seconds.

An interval of day-time consists of +days hours:months:seconds.fractional with values ranging from-999999 23:59:59.999999999 to +999999 23:59:59.999999999. The value representation is the samefor all types of resolutions. For example, an interval of seconds of 70 is always represented inan interval-of-days-to-seconds format (with default precisions): +00 00:01:10.000000.

Declaration

INTERVAL DAY
INTERVAL DAY(p1)
INTERVAL DAY(p1) TO HOUR
INTERVAL DAY(p1) TO MINUTE
INTERVAL DAY(p1) TO SECOND(p2)
INTERVAL HOUR
INTERVAL HOUR TO MINUTE
INTERVAL HOUR TO SECOND(p2)
INTERVAL MINUTE
INTERVAL MINUTE TO SECOND(p2)
INTERVAL SECOND
INTERVAL SECOND(p2)

DataTypes.INTERVAL(DataTypes.DAY())
DataTypes.INTERVAL(DataTypes.DAY(p1))
DataTypes.INTERVAL(DataTypes.DAY(p1), DataTypes.HOUR())
DataTypes.INTERVAL(DataTypes.DAY(p1), DataTypes.MINUTE())
DataTypes.INTERVAL(DataTypes.DAY(p1), DataTypes.SECOND(p2))
DataTypes.INTERVAL(DataTypes.HOUR())
DataTypes.INTERVAL(DataTypes.HOUR(), DataTypes.MINUTE())
DataTypes.INTERVAL(DataTypes.HOUR(), DataTypes.SECOND(p2))
DataTypes.INTERVAL(DataTypes.MINUTE())
DataTypes.INTERVAL(DataTypes.MINUTE(), DataTypes.SECOND(p2))
DataTypes.INTERVAL(DataTypes.SECOND())
DataTypes.INTERVAL(DataTypes.SECOND(p2))

The type can be declared using the above combinations where p1 is the number of digits of days(day precision) and p2 is the number of digits of fractional seconds (fractional precision).p1 must have a value between 1 and 6 (both inclusive). p2 must have a value between 0and 9 (both inclusive). If no p1 is specified, it is equal to 2 by default. If no p2 isspecified, it is equal to 6 by default.

Bridging to JVM Types

Constructured Data Types

ARRAY

Data type of an array of elements with same subtype.

Compared to the SQL standard, the maximum cardinality of an array cannot be specified but isfixed at 2,147,483,647. Also, any valid type is supported as a subtype.

Declaration

ARRAY<t>
t ARRAY

DataTypes.ARRAY(t)

The type can be declared using ARRAY<t> where t is the data type of the containedelements.

t ARRAY is a synonym for being closer to the SQL standard. For example, INT ARRAY isequivalent to ARRAY<INT>.

Bridging to JVM Types

MULTISET

Data type of a multiset (=bag). Unlike a set, it allows for multiple instances for each of itselements with a common subtype. Each unique value (including NULL) is mapped to some multiplicity.

There is no restriction of element types; it is the responsibility of the user to ensure uniqueness.

Declaration

MULTISET<t>
t MULTISET

DataTypes.MULTISET(t)

The type can be declared using MULTISET<t> where t is the data typeof the contained elements.

t MULTISET is a synonym for being closer to the SQL standard. For example, INT MULTISET isequivalent to MULTISET<INT>.

Bridging to JVM Types

ROW

Data type of a sequence of fields.

A field consists of a field name, field type, and an optional description. The most specific typeof a row of a table is a row type. In this case, each column of the row corresponds to the fieldof the row type that has the same ordinal position as the column.

Compared to the SQL standard, an optional field description simplifies the handling with complexstructures.

A row type is similar to the STRUCT type known from other non-standard-compliant frameworks.

Declaration

ROW<n0 t0, n1 t1, ...>
ROW<n0 t0 'd0', n1 t1 'd1', ...>
ROW(n0 t0, n1 t1, ...>
ROW(n0 t0 'd0', n1 t1 'd1', ...)

DataTypes.ROW(DataTypes.FIELD(n0, t0), DataTypes.FIELD(n1, t1), ...)
DataTypes.ROW(DataTypes.FIELD(n0, t0, d0), DataTypes.FIELD(n1, t1, d1), ...)

The type can be declared using ROW<n0 t0 'd0', n1 t1 'd1', …> where n is the unique name ofa field, t is the logical type of a field, d is the description of a field.

ROW(…) is a synonym for being closer to the SQL standard. For example, ROW(myField INT, myOtherField BOOLEAN) isequivalent to ROW<myField INT, myOtherField BOOLEAN>.

Bridging to JVM Types

Other Data Types

BOOLEAN

Data type of a boolean with a (possibly) three-valued logic of TRUE, FALSE, and UNKNOWN.

Declaration

BOOLEAN

DataTypes.BOOLEAN()

Bridging to JVM Types

NULL

Data type for representing untyped NULL values.

The null type is an extension to the SQL standard. A null type has no other valueexcept NULL, thus, it can be cast to any nullable type similar to JVM semantics.

This type helps in representing unknown types in API calls that use a NULL literalas well as bridging to formats such as JSON or Avro that define such a type as well.

This type is not very useful in practice and is just mentioned here for completeness.

Declaration

NULL

DataTypes.NULL()

Bridging to JVM Types

ANY

Data type of an arbitrary serialized type. This type is a black box within the table ecosystemand is only deserialized at the edges.

The any type is an extension to the SQL standard.

Declaration

ANY('class', 'snapshot')

DataTypes.ANY(class, serializer)
DataTypes.ANY(typeInfo)

The type can be declared using ANY('class', 'snapshot') where class is the originating class andsnapshot is the serialized TypeSerializerSnapshot in Base64 encoding. Usually, the type string is notdeclared directly but is generated while persisting the type.

In the API, the ANY type can be declared either by directly supplying a Class + TypeSerializer orby passing TypeInformation and let the framework extract Class + TypeSerializer from there.

Bridging to JVM Types