Time Attributes

Flink is able to process streaming data based on different notions of time.

  • Processing time refers to the system time of the machine (also known as “wall-clock time”) that is executing the respective operation.
  • Event time refers to the processing of streaming data based on timestamps which are attached to each row. The timestamps can encode when an event happened.
  • Ingestion time is the time that events enter Flink; internally, it is treated similarly to event time.

For more information about time handling in Flink, see the introduction about Event Time and Watermarks.

This page explains how time attributes can be defined for time-based operations in Flink’s Table API & SQL.

Introduction to Time Attributes

Time-based operations such as windows in both the Table API and SQL require information about the notion of time and its origin. Therefore, tables can offer logical time attributes for indicating time and accessing corresponding timestamps in table programs.

Time attributes can be part of every table schema. They are defined when creating a table from a CREATE TABLE DDL or a DataStream or are pre-defined when using a TableSource. Once a time attribute has been defined at the beginning, it can be referenced as a field and can be used in time-based operations.

As long as a time attribute is not modified and is simply forwarded from one part of the query to another, it remains a valid time attribute. Time attributes behave like regular timestamps and can be accessed for calculations. If a time attribute is used in a calculation, it will be materialized and becomes a regular timestamp. Regular timestamps do not cooperate with Flink’s time and watermarking system and thus can not be used for time-based operations anymore.

Table programs require that the corresponding time characteristic has been specified for the streaming environment:

  1. final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
  3. env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime); // default
  5. // alternatively:
  6. // env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
  7. // env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
  1. val env = StreamExecutionEnvironment.getExecutionEnvironment
  3. env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime) // default
  5. // alternatively:
  6. // env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime)
  7. // env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
  1. env = StreamExecutionEnvironment.get_execution_environment()
  3. env.set_stream_time_characteristic(TimeCharacteristic.ProcessingTime)  # default
  5. # alternatively:
  6. # env.set_stream_time_characteristic(TimeCharacteristic.IngestionTime)
  7. # env.set_stream_time_characteristic(TimeCharacteristic.EventTime)

Processing time

Processing time allows a table program to produce results based on the time of the local machine. It is the simplest notion of time but does not provide determinism. It neither requires timestamp extraction nor watermark generation.

There are three ways to define a processing time attribute.

Defining in create table DDL

The processing time attribute is defined as a computed column in create table DDL using the system PROCTIME() function. Please see CREATE TABLE DDL for more information about computed column.

  1. CREATE TABLE user_actions (
  2.   user_name STRING,
  3.   data STRING,
  4.   user_action_time AS PROCTIME() -- declare an additional field as a processing time attribute
  5. ) WITH (
  6.   ...
  7. );
  9. SELECT TUMBLE_START(user_action_time, INTERVAL '10' MINUTE), COUNT(DISTINCT user_name)
  10. FROM user_actions
  11. GROUP BY TUMBLE(user_action_time, INTERVAL '10' MINUTE);

During DataStream-to-Table Conversion

The processing time attribute is defined with the .proctime property during schema definition. The time attribute must only extend the physical schema by an additional logical field. Thus, it can only be defined at the end of the schema definition.

  1. DataStream<Tuple2<String, String>> stream = ...;
  3. // declare an additional logical field as a processing time attribute
  4. Table table = tEnv.fromDataStream(stream, "user_name, data, user_action_time.proctime");
  6. WindowedTable windowedTable = table.window(Tumble.over("10.minutes").on("user_action_time").as("userActionWindow"));
  1. val stream: DataStream[(String, String)] = ...
  3. // declare an additional logical field as a processing time attribute
  4. val table = tEnv.fromDataStream(stream, 'UserActionTimestamp, 'user_name, 'data, 'user_action_time.proctime)
  6. val windowedTable = table.window(Tumble over 10.minutes on 'user_action_time as 'userActionWindow)

Using a TableSource

The processing time attribute is defined by a TableSource that implements the DefinedProctimeAttribute interface. The logical time attribute is appended to the physical schema defined by the return type of the TableSource.

  1. // define a table source with a processing attribute
  2. public class UserActionSource implements StreamTableSource<Row>, DefinedProctimeAttribute {
  4.     @Override
  5.     public TypeInformation<Row> getReturnType() {
  6.         String[] names = new String[] {"user_name" , "data"};
  7.         TypeInformation[] types = new TypeInformation[] {Types.STRING(), Types.STRING()};
  8.         return Types.ROW(names, types);
  9.     }
  11.     @Override
  12.     public DataStream<Row> getDataStream(StreamExecutionEnvironment execEnv) {
  13.         // create stream
  14.         DataStream<Row> stream = ...;
  15.         return stream;
  16.     }
  18.     @Override
  19.     public String getProctimeAttribute() {
  20.         // field with this name will be appended as a third field
  21.         return "user_action_time";
  22.     }
  23. }
  25. // register table source
  26. tEnv.registerTableSource("user_actions", new UserActionSource());
  28. WindowedTable windowedTable = tEnv
  29.     .from("user_actions")
  30.     .window(Tumble.over("10.minutes").on("user_action_time").as("userActionWindow"));
  1. // define a table source with a processing attribute
  2. class UserActionSource extends StreamTableSource[Row] with DefinedProctimeAttribute {
  4.     override def getReturnType = {
  5.         val names = Array[String]("user_name" , "data")
  6.         val types = Array[TypeInformation[_]](Types.STRING, Types.STRING)
  7.         Types.ROW(names, types)
  8.     }
  10.     override def getDataStream(execEnv: StreamExecutionEnvironment): DataStream[Row] = {
  11.         // create stream
  12.         val stream = ...
  13.         stream
  14.     }
  16.     override def getProctimeAttribute = {
  17.         // field with this name will be appended as a third field
  18.         "user_action_time"
  19.     }
  20. }
  22. // register table source
  23. tEnv.registerTableSource("user_actions", new UserActionSource)
  25. val windowedTable = tEnv
  26.     .from("user_actions")
  27.     .window(Tumble over 10.minutes on 'user_action_time as 'userActionWindow)

Event time

Event time allows a table program to produce results based on the time that is contained in every record. This allows for consistent results even in case of out-of-order events or late events. It also ensures replayable results of the table program when reading records from persistent storage.

Additionally, event time allows for unified syntax for table programs in both batch and streaming environments. A time attribute in a streaming environment can be a regular field of a record in a batch environment.

In order to handle out-of-order events and distinguish between on-time and late events in streaming, Flink needs to extract timestamps from events and make some kind of progress in time (so-called watermarks).

An event time attribute can be defined either in create table DDL or during DataStream-to-Table conversion or by using a TableSource.

Defining in create table DDL

The event time attribute is defined using WATERMARK statement in CREATE TABLE DDL. A watermark statement defines a watermark generation expression on an existing event time field, which marks the event time field as event time attribute. Please see CREATE TABLE DDL for more information about watermark statement and watermark strategies.

  1. CREATE TABLE user_actions (
  2.   user_name STRING,
  3.   data STRING,
  4.   user_action_time TIMESTAMP(3),
  5.   -- declare user_action_time as event time attribute and use 5 seconds delayed watermark strategy
  6.   WATERMARK FOR user_action_time AS user_action_time - INTERVAL '5' SECOND
  7. ) WITH (
  8.   ...
  9. );
  11. SELECT TUMBLE_START(user_action_time, INTERVAL '10' MINUTE), COUNT(DISTINCT user_name)
  12. FROM user_actions
  13. GROUP BY TUMBLE(user_action_time, INTERVAL '10' MINUTE);

During DataStream-to-Table Conversion

The event time attribute is defined with the .rowtime property during schema definition. Timestamps and watermarks must have been assigned in the DataStream that is converted.

There are two ways of defining the time attribute when converting a DataStream into a Table. Depending on whether the specified .rowtime field name exists in the schema of the DataStream or not, the timestamp field is either

  • appended as a new field to the schema or
  • replaces an existing field.

In either case the event time timestamp field will hold the value of the DataStream event time timestamp.

  1. // Option 1:
  3. // extract timestamp and assign watermarks based on knowledge of the stream
  4. DataStream<Tuple2<String, String>> stream = inputStream.assignTimestampsAndWatermarks(...);
  6. // declare an additional logical field as an event time attribute
  7. Table table = tEnv.fromDataStream(stream, "user_name, data, user_action_time.rowtime");
  10. // Option 2:
  12. // extract timestamp from first field, and assign watermarks based on knowledge of the stream
  13. DataStream<Tuple3<Long, String, String>> stream = inputStream.assignTimestampsAndWatermarks(...);
  15. // the first field has been used for timestamp extraction, and is no longer necessary
  16. // replace first field with a logical event time attribute
  17. Table table = tEnv.fromDataStream(stream, "user_action_time.rowtime, user_name, data");
  19. // Usage:
  21. WindowedTable windowedTable = table.window(Tumble.over("10.minutes").on("user_action_time").as("userActionWindow"));
  1. // Option 1:
  3. // extract timestamp and assign watermarks based on knowledge of the stream
  4. val stream: DataStream[(String, String)] = inputStream.assignTimestampsAndWatermarks(...)
  6. // declare an additional logical field as an event time attribute
  7. val table = tEnv.fromDataStream(stream, 'user_name, 'data, 'user_action_time.rowtime)
  10. // Option 2:
  12. // extract timestamp from first field, and assign watermarks based on knowledge of the stream
  13. val stream: DataStream[(Long, String, String)] = inputStream.assignTimestampsAndWatermarks(...)
  15. // the first field has been used for timestamp extraction, and is no longer necessary
  16. // replace first field with a logical event time attribute
  17. val table = tEnv.fromDataStream(stream, 'user_action_time.rowtime, 'user_name, 'data)
  19. // Usage:
  21. val windowedTable = table.window(Tumble over 10.minutes on 'user_action_time as 'userActionWindow)

Using a TableSource

The event time attribute is defined by a TableSource that implements the DefinedRowtimeAttributes interface. The getRowtimeAttributeDescriptors() method returns a list of RowtimeAttributeDescriptor for describing the final name of a time attribute, a timestamp extractor to derive the values of the attribute, and the watermark strategy associated with the attribute.

Please make sure that the DataStream returned by the getDataStream() method is aligned with the defined time attribute.The timestamps of the DataStream (the ones which are assigned by a TimestampAssigner) are only considered if a StreamRecordTimestamp timestamp extractor is defined.Watermarks of a DataStream are only preserved if a PreserveWatermarks watermark strategy is defined.Otherwise, only the values of the TableSource’s rowtime attribute are relevant.

  1. // define a table source with a rowtime attribute
  2. public class UserActionSource implements StreamTableSource<Row>, DefinedRowtimeAttributes {
  4.     @Override
  5.     public TypeInformation<Row> getReturnType() {
  6.         String[] names = new String[] {"user_name", "data", "user_action_time"};
  7.         TypeInformation[] types =
  8.             new TypeInformation[] {Types.STRING(), Types.STRING(), Types.LONG()};
  9.         return Types.ROW(names, types);
  10.     }
  12.     @Override
  13.     public DataStream<Row> getDataStream(StreamExecutionEnvironment execEnv) {
  14.         // create stream
  15.         // ...
  16.         // assign watermarks based on the "user_action_time" attribute
  17.         DataStream<Row> stream = inputStream.assignTimestampsAndWatermarks(...);
  18.         return stream;
  19.     }
  21.     @Override
  22.     public List<RowtimeAttributeDescriptor> getRowtimeAttributeDescriptors() {
  23.         // Mark the "user_action_time" attribute as event-time attribute.
  24.         // We create one attribute descriptor of "user_action_time".
  25.         RowtimeAttributeDescriptor rowtimeAttrDescr = new RowtimeAttributeDescriptor(
  26.             "user_action_time",
  27.             new ExistingField("user_action_time"),
  28.             new AscendingTimestamps());
  29.         List<RowtimeAttributeDescriptor> listRowtimeAttrDescr = Collections.singletonList(rowtimeAttrDescr);
  30.         return listRowtimeAttrDescr;
  31.     }
  32. }
  34. // register the table source
  35. tEnv.registerTableSource("user_actions", new UserActionSource());
  37. WindowedTable windowedTable = tEnv
  38.     .from("user_actions")
  39.     .window(Tumble.over("10.minutes").on("user_action_time").as("userActionWindow"));
  1. // define a table source with a rowtime attribute
  2. class UserActionSource extends StreamTableSource[Row] with DefinedRowtimeAttributes {
  4.     override def getReturnType = {
  5.         val names = Array[String]("user_name" , "data", "user_action_time")
  6.         val types = Array[TypeInformation[_]](Types.STRING, Types.STRING, Types.LONG)
  7.         Types.ROW(names, types)
  8.     }
  10.     override def getDataStream(execEnv: StreamExecutionEnvironment): DataStream[Row] = {
  11.         // create stream
  12.         // ...
  13.         // assign watermarks based on the "user_action_time" attribute
  14.         val stream = inputStream.assignTimestampsAndWatermarks(...)
  15.         stream
  16.     }
  18.     override def getRowtimeAttributeDescriptors: util.List[RowtimeAttributeDescriptor] = {
  19.         // Mark the "user_action_time" attribute as event-time attribute.
  20.         // We create one attribute descriptor of "user_action_time".
  21.         val rowtimeAttrDescr = new RowtimeAttributeDescriptor(
  22.             "user_action_time",
  23.             new ExistingField("user_action_time"),
  24.             new AscendingTimestamps)
  25.         val listRowtimeAttrDescr = Collections.singletonList(rowtimeAttrDescr)
  26.         listRowtimeAttrDescr
  27.     }
  28. }
  30. // register the table source
  31. tEnv.registerTableSource("user_actions", new UserActionSource)
  33. val windowedTable = tEnv
  34.     .from("user_actions")
  35.     .window(Tumble over 10.minutes on 'user_action_time as 'userActionWindow)