Side Outputs

In addition to the main stream that results from DataStream operations, you can also produce any number of additional side output result streams. The type of data in the result streams does not have to match the type of data in the main stream and the types of the different side outputs can also differ. This operation can be useful when you want to split a stream of data where you would normally have to replicate the stream and then filter out from each stream the data that you don’t want to have.

When using side outputs, you first need to define an OutputTag that will be used to identify a side output stream:

Java

  1. // this needs to be an anonymous inner class, so that we can analyze the type
  2. OutputTag<String> outputTag = new OutputTag<String>("side-output") {};

Scala

  1. val outputTag = OutputTag[String]("side-output")

Python

  1. output_tag = OutputTag("side-output", Types.STRING())

Notice how the OutputTag is typed according to the type of elements that the side output stream contains.

Emitting data to a side output is possible from the following functions:

You can use the Context parameter, which is exposed to users in the above functions, to emit data to a side output identified by an OutputTag. Here is an example of emitting side output data from a ProcessFunction:

Java

  1. DataStream<Integer> input = ...;
  3. final OutputTag<String> outputTag = new OutputTag<String>("side-output"){};
  5. SingleOutputStreamOperator<Integer> mainDataStream = input
  6.   .process(new ProcessFunction<Integer, Integer>() {
  8.       @Override
  9.       public void processElement(
  10.           Integer value,
  11.           Context ctx,
  12.           Collector<Integer> out) throws Exception {
  13.         // emit data to regular output
  14.         out.collect(value);
  16.         // emit data to side output
  17.         ctx.output(outputTag, "sideout-" + String.valueOf(value));
  18.       }
  19.     });

Scala

  2. val input: DataStream[Int] = ...
  3. val outputTag = OutputTag[String]("side-output")
  5. val mainDataStream = input
  6.   .process(new ProcessFunction[Int, Int] {
  7.     override def processElement(
  8.         value: Int,
  9.         ctx: ProcessFunction[Int, Int]#Context,
  10.         out: Collector[Int]): Unit = {
  11.       // emit data to regular output
  12.       out.collect(value)
  14.       // emit data to side output
  15.       ctx.output(outputTag, "sideout-" + String.valueOf(value))
  16.     }
  17.   })

Python

  1. input = ...  # type: DataStream
  2. output_tag = OutputTag("side-output", Types.STRING())
  4. class MyProcessFunction(ProcessFunction):
  6.     def process_element(self, value: int, ctx: ProcessFunction.Context):
  7.         # emit data to regular output
  8.         yield value
  10.         # emit data to side output
  11.         yield output_tag, "sideout-" + str(value)
  14. main_data_stream = input \
  15.     .process(MyProcessFunction(), Types.INT())

For retrieving the side output stream you use getSideOutput(OutputTag) on the result of the DataStream operation. This will give you a DataStream that is typed to the result of the side output stream:

Java

  1. final OutputTag<String> outputTag = new OutputTag<String>("side-output"){};
  3. SingleOutputStreamOperator<Integer> mainDataStream = ...;
  5. DataStream<String> sideOutputStream = mainDataStream.getSideOutput(outputTag);

Scala

  1. val outputTag = OutputTag[String]("side-output")
  3. val mainDataStream = ...
  5. val sideOutputStream: DataStream[String] = mainDataStream.getSideOutput(outputTag)

Python

  1. output_tag = OutputTag("side-output", Types.STRING())
  3. main_data_stream = ...  # type: DataStream
  5. side_output_stream = main_data_stream.get_side_output(output_tag)  # type: DataStream

Note If it produces side output, get_side_output(OutputTag) must be called in Python API. Otherwise, the result of side output stream will be output into the main stream which is unexpected and may fail the job when the data types are different.