Side Outputs

In addition to the main stream that results from DataStream operations, you can also produce anynumber of additional side output result streams. The type of data in the result streams does nothave to match the type of data in the main stream and the types of the different side outputs canalso differ. This operation can be useful when you want to split a stream of data where you wouldnormally have to replicate the stream and then filter out from each stream the data that you don’twant to have.

When using side outputs, you first need to define an OutputTag that will be used to identify aside output stream:

  1. // this needs to be an anonymous inner class, so that we can analyze the type
  2. OutputTag<String> outputTag = new OutputTag<String>("side-output") {};
  1. val outputTag = OutputTag[String]("side-output")

Notice how the OutputTag is typed according to the type of elements that the side output streamcontains.

Emitting data to a side output is possible from the following functions:

  • ProcessFunction
  • KeyedProcessFunction
  • CoProcessFunction
  • KeyedCoProcessFunction
  • ProcessWindowFunction
  • ProcessAllWindowFunctionYou can use the Context parameter, which is exposed to users in the above functions, to emitdata to a side output identified by an OutputTag. Here is an example of emitting side outputdata from a ProcessFunction:
  1. DataStream<Integer> input = ...;
  3. final OutputTag<String> outputTag = new OutputTag<String>("side-output"){};
  5. SingleOutputStreamOperator<Integer> mainDataStream = input
  6.   .process(new ProcessFunction<Integer, Integer>() {
  8.       @Override
  9.       public void processElement(
  10.           Integer value,
  11.           Context ctx,
  12.           Collector<Integer> out) throws Exception {
  13.         // emit data to regular output
  14.         out.collect(value);
  16.         // emit data to side output
  17.         ctx.output(outputTag, "sideout-" + String.valueOf(value));
  18.       }
  19.     });
  1. val input: DataStream[Int] = ...
  2. val outputTag = OutputTag[String]("side-output")
  4. val mainDataStream = input
  5.   .process(new ProcessFunction[Int, Int] {
  6.     override def processElement(
  7.         value: Int,
  8.         ctx: ProcessFunction[Int, Int]#Context,
  9.         out: Collector[Int]): Unit = {
  10.       // emit data to regular output
  11.       out.collect(value)
  13.       // emit data to side output
  14.       ctx.output(outputTag, "sideout-" + String.valueOf(value))
  15.     }
  16.   })

For retrieving the side output stream you use getSideOutput(OutputTag)on the result of the DataStream operation. This will give you a DataStream that is typedto the result of the side output stream:

  1. final OutputTag<String> outputTag = new OutputTag<String>("side-output"){};
  3. SingleOutputStreamOperator<Integer> mainDataStream = ...;
  5. DataStream<String> sideOutputStream = mainDataStream.getSideOutput(outputTag);
  1. val outputTag = OutputTag[String]("side-output")
  3. val mainDataStream = ...
  5. val sideOutputStream: DataStream[String] = mainDataStream.getSideOutput(outputTag)