算子

用户通过算子能将一个或多个 DataStream 转换成新的 DataStream，在应用程序中可以将多个数据转换算子合并成一个复杂的数据流拓扑。

这部分内容将描述 Flink DataStream API 中基本的数据转换API，数据转换后各种数据分区方式，以及算子的链接策略。

数据流转换

Map

DataStream → DataStream

Takes one element and produces one element. A map function that doubles the values of the input stream:

Java

  1. DataStream<Integer> dataStream = //...
  2. dataStream.map(new MapFunction<Integer, Integer>() {
  3.     @Override
  4.     public Integer map(Integer value) throws Exception {
  5.         return 2 * value;
  6.     }
  7. });

Scala

  1. dataStream.map { x => x * 2 }

Python

  1. data_stream = env.from_collection(collection=[1, 2, 3, 4, 5])
  2. data_stream.map(lambda x: 2 * x, output_type=Types.INT())

FlatMap

DataStream → DataStream

Takes one element and produces zero, one, or more elements. A flatmap function that splits sentences to words:

Java

  1. dataStream.flatMap(new FlatMapFunction<String, String>() {
  2.     @Override
  3.     public void flatMap(String value, Collector<String> out)
  4.         throws Exception {
  5.         for(String word: value.split(" ")){
  6.             out.collect(word);
  7.         }
  8.     }
  9. });

Scala

  1. dataStream.flatMap { str => str.split(" ") }

Python

  1. data_stream = env.from_collection(collection=['hello apache flink', 'streaming compute'])
  2. data_stream.flat_map(lambda x: x.split(' '), output_type=Types.STRING())

Filter

DataStream → DataStream

Evaluates a boolean function for each element and retains those for which the function returns true. A filter that filters out zero values:

Java

  1. dataStream.filter(new FilterFunction<Integer>() {
  2.     @Override
  3.     public boolean filter(Integer value) throws Exception {
  4.         return value != 0;
  5.     }
  6. });

Scala

  1. dataStream.filter { _ != 0 }

Python

  1. data_stream = env.from_collection(collection=[0, 1, 2, 3, 4, 5])
  2. data_stream.filter(lambda x: x != 0)

KeyBy

DataStream → KeyedStream

Logically partitions a stream into disjoint partitions. All records with the same key are assigned to the same partition. Internally, keyBy() is implemented with hash partitioning. There are different ways to specify keys.

Java

  1. dataStream.keyBy(value -> value.getSomeKey());
  2. dataStream.keyBy(value -> value.f0);

Scala

  1. dataStream.keyBy(_.someKey)
  2. dataStream.keyBy(_._1)

Python

  1. data_stream = env.from_collection(collection=[(1, 'a'), (2, 'a'), (3, 'b')])
  2. data_stream.key_by(lambda x: x[1], key_type=Types.STRING()) // Key by the result of KeySelector

A type cannot be a key if:

  1. it is a POJO type but does not override the hashCode() method and relies on the Object.hashCode() implementation.
  2. it is an array of any type.

Reduce

KeyedStream → DataStream

A “rolling” reduce on a keyed data stream. Combines the current element with the last reduced value and emits the new value.

A reduce function that creates a stream of partial sums:

Java

  1. keyedStream.reduce(new ReduceFunction<Integer>() {
  2.     @Override
  3.     public Integer reduce(Integer value1, Integer value2)
  4.     throws Exception {
  5.         return value1 + value2;
  6.     }
  7. });

Scala

  1. keyedStream.reduce { _ + _ }

Python

  1. data_stream = env.from_collection(collection=[(1, 'a'), (2, 'a'), (3, 'a'), (4, 'b')], type_info=Types.TUPLE([Types.INT(), Types.STRING()]))
  2. data_stream.key_by(lambda x: x[1]).reduce(lambda a, b: (a[0] + b[0], b[1]))

Window

KeyedStream → WindowedStream

Windows can be defined on already partitioned KeyedStreams. Windows group the data in each key according to some characteristic (e.g., the data that arrived within the last 5 seconds). See windows for a complete description of windows.

Java

  1. dataStream
  2.   .keyBy(value -> value.f0)
  3.   .window(TumblingEventTimeWindows.of(Time.seconds(5)));

Scala

  1. dataStream
  2.   .keyBy(_._1)
  3.   .window(TumblingEventTimeWindows.of(Time.seconds(5)))

Python

This feature is not yet supported in Python

WindowAll

DataStream → AllWindowedStream

Windows can be defined on regular DataStreams. Windows group all the stream events according to some characteristic (e.g., the data that arrived within the last 5 seconds). See windows for a complete description of windows.

This is in many cases a non-parallel transformation. All records will be gathered in one task for the windowAll operator.

Java

  1. dataStream
  2.   .windowAll(TumblingEventTimeWindows.of(Time.seconds(5)));

Scala

  1. dataStream
  2.   .windowAll(TumblingEventTimeWindows.of(Time.seconds(5)))

Python

This feature is not yet supported in Python

Window Apply

WindowedStream → DataStream

AllWindowedStream → DataStream

Applies a general function to the window as a whole. Below is a function that manually sums the elements of a window.

If you are using a windowAll transformation, you need to use an AllWindowFunction instead.

Java

  1. windowedStream.apply(new WindowFunction<Tuple2<String,Integer>, Integer, Tuple, Window>() {
  2.     public void apply (Tuple tuple,
  3.             Window window,
  4.             Iterable<Tuple2<String, Integer>> values,
  5.             Collector<Integer> out) throws Exception {
  6.         int sum = 0;
  7.         for (value t: values) {
  8.             sum += t.f1;
  9.         }
  10.         out.collect (new Integer(sum));
  11.     }
  12. });
  14. // applying an AllWindowFunction on non-keyed window stream
  15. allWindowedStream.apply (new AllWindowFunction<Tuple2<String,Integer>, Integer, Window>() {
  16.     public void apply (Window window,
  17.             Iterable<Tuple2<String, Integer>> values,
  18.             Collector<Integer> out) throws Exception {
  19.         int sum = 0;
  20.         for (value t: values) {
  21.             sum += t.f1;
  22.         }
  23.         out.collect (new Integer(sum));
  24.     }
  25. });

Scala

  1. windowedStream.apply { WindowFunction }
  3. // applying an AllWindowFunction on non-keyed window stream
  4. allWindowedStream.apply { AllWindowFunction }

Python

This feature is not yet supported in Python

WindowReduce

WindowedStream → DataStream

Applies a functional reduce function to the window and returns the reduced value.

Java

  1. windowedStream.reduce (new ReduceFunction<Tuple2<String,Integer>>() {
  2.     public Tuple2<String, Integer> reduce(Tuple2<String, Integer> value1, Tuple2<String, Integer> value2) throws Exception {
  3.         return new Tuple2<String,Integer>(value1.f0, value1.f1 + value2.f1);
  4.     }
  5. });

Scala

  1. windowedStream.reduce { _ + _ }

Python

This feature is not yet supported in Python

Union

DataStream* → DataStream

Union of two or more data streams creating a new stream containing all the elements from all the streams. Note: If you union a data stream with itself you will get each element twice in the resulting stream.

Java

  1. dataStream.union(otherStream1, otherStream2, ...);

Scala

  1. dataStream.union(otherStream1, otherStream2, ...);

Python

  1. data_stream.union(otherStream1, otherStream2, ...)

Window Join

DataStream,DataStream → DataStream

Join two data streams on a given key and a common window.

Java

  1. dataStream.join(otherStream)
  2.     .where(<key selector>).equalTo(<key selector>)
  3.     .window(TumblingEventTimeWindows.of(Time.seconds(3)))
  4.     .apply (new JoinFunction () {...});

Scala

  1. dataStream.join(otherStream)
  2.     .where(<key selector>).equalTo(<key selector>)
  3.     .window(TumblingEventTimeWindows.of(Time.seconds(3)))
  4.     .apply { ... }

Python

This feature is not yet supported in Python

Interval Join

KeyedStream,KeyedStream → DataStream

Join two elements e1 and e2 of two keyed streams with a common key over a given time interval, so that e1.timestamp + lowerBound <= e2.timestamp <= e1.timestamp + upperBound.

Java

  1. // this will join the two streams so that
  2. // key1 == key2 && leftTs - 2 < rightTs < leftTs + 2
  3. keyedStream.intervalJoin(otherKeyedStream)
  4.     .between(Time.milliseconds(-2), Time.milliseconds(2)) // lower and upper bound
  5.     .upperBoundExclusive(true) // optional
  6.     .lowerBoundExclusive(true) // optional
  7.     .process(new IntervalJoinFunction() {...});

Scala

  1. // this will join the two streams so that
  2. // key1 == key2 && leftTs - 2 < rightTs < leftTs + 2
  3. keyedStream.intervalJoin(otherKeyedStream)
  4.     .between(Time.milliseconds(-2), Time.milliseconds(2)) 
  5.     // lower and upper bound
  6.     .upperBoundExclusive(true) // optional
  7.     .lowerBoundExclusive(true) // optional
  8.     .process(new IntervalJoinFunction() {...})

Python

This feature is not yet supported in Python

Window CoGroup

DataStream,DataStream → DataStream

Cogroups two data streams on a given key and a common window.

Java

  1. dataStream.coGroup(otherStream)
  2.     .where(0).equalTo(1)
  3.     .window(TumblingEventTimeWindows.of(Time.seconds(3)))
  4.     .apply (new CoGroupFunction () {...});

Scala

  1. dataStream.coGroup(otherStream)
  2.     .where(0).equalTo(1)
  3.     .window(TumblingEventTimeWindows.of(Time.seconds(3)))
  4.     .apply {}

Python

This feature is not yet supported in Python

Connect

DataStream,DataStream → ConnectedStream

“Connects” two data streams retaining their types. Connect allowing for shared state between the two streams.

Java

  1. DataStream<Integer> someStream = //...
  2. DataStream<String> otherStream = //...
  4. ConnectedStreams<Integer, String> connectedStreams = someStream.connect(otherStream);

Scala

  1. someStream : DataStream[Int] = ...
  2. otherStream : DataStream[String] = ...
  4. val connectedStreams = someStream.connect(otherStream)

Python

  1. stream_1 = ...
  2. stream_2 = ...
  3. connected_streams = stream_1.connect(stream_2)

CoMap, CoFlatMap

ConnectedStream → DataStream

Similar to map and flatMap on a connected data stream

Java

  1. connectedStreams.map(new CoMapFunction<Integer, String, Boolean>() {
  2.     @Override
  3.     public Boolean map1(Integer value) {
  4.         return true;
  5.     }
  7.     @Override
  8.     public Boolean map2(String value) {
  9.         return false;
  10.     }
  11. });
  12. connectedStreams.flatMap(new CoFlatMapFunction<Integer, String, String>() {
  14.    @Override
  15.    public void flatMap1(Integer value, Collector<String> out) {
  16.        out.collect(value.toString());
  17.    }
  19.    @Override
  20.    public void flatMap2(String value, Collector<String> out) {
  21.        for (String word: value.split(" ")) {
  22.          out.collect(word);
  23.        }
  24.    }
  25. });

Scala

  1. connectedStreams.map(
  2.     (_ : Int) => true,
  3.     (_ : String) => false
  4. )
  5. connectedStreams.flatMap(
  6.     (_ : Int) => true,
  7.     (_ : String) => false
  8. )

Python

  1. class MyCoMapFunction(CoMapFunction):
  3.     def map1(self, value):
  4.         return value[0] + 1, value[1]
  6.     def map2(self, value):
  7.         return value[0], value[1] + 'flink'
  9. class MyCoFlatMapFunction(CoFlatMapFunction):
  11.     def flat_map1(self, value)
  12.         for i in range(value[0]):
  13.             yield i
  15.     def flat_map2(self, value):
  16.         yield value[0] + 1
  18. connectedStreams.map(MyCoMapFunction())
  19. connectedStreams.flat_map(MyCoFlatMapFunction())

Iterate

DataStream → IterativeStream → ConnectedStream

Creates a “feedback” loop in the flow, by redirecting the output of one operator to some previous operator. This is especially useful for defining algorithms that continuously update a model. The following code starts with a stream and applies the iteration body continuously. Elements that are greater than 0 are sent back to the feedback channel, and the rest of the elements are forwarded downstream.

Java

  1. IterativeStream<Long> iteration = initialStream.iterate();
  2. DataStream<Long> iterationBody = iteration.map (/*do something*/);
  3. DataStream<Long> feedback = iterationBody.filter(new FilterFunction<Long>(){
  4.     @Override
  5.     public boolean filter(Long value) throws Exception {
  6.         return value > 0;
  7.     }
  8. });
  9. iteration.closeWith(feedback);
  10. DataStream<Long> output = iterationBody.filter(new FilterFunction<Long>(){
  11.     @Override
  12.     public boolean filter(Long value) throws Exception {
  13.         return value <= 0;
  14.     }
  15. });

Scala

  1. initialStream.iterate {
  2.   iteration => {
  3.     val iterationBody = iteration.map {/*do something*/}
  4.     (iterationBody.filter(_ > 0), iterationBody.filter(_ <= 0))
  5.   }
  6. }

Python

This feature is not yet supported in Python

物理分区

Flink 也提供以下方法让用户根据需要在数据转换完成后对数据分区进行更细粒度的配置。

Custom Partitioning

DataStream → DataStream

Uses a user-defined Partitioner to select the target task for each element.

Java

  1. dataStream.partitionCustom(partitioner, "someKey");
  2. dataStream.partitionCustom(partitioner, 0);

Scala

  1. dataStream.partitionCustom(partitioner, "someKey")
  2. dataStream.partitionCustom(partitioner, 0)

Python

  1. data_stream = env.from_collection(collection=[(2, 'a'), (2, 'a'), (3, 'b')])
  2. data_stream.partition_custom(lambda key, num_partition: key % partition, lambda x: x[0])

Random Partitioning

DataStream → DataStream

Partitions elements randomly according to a uniform distribution.

Java

  1. dataStream.shuffle();

Scala

  1. dataStream.shuffle()

Python

  1. data_stream.shuffle()

Rescaling

DataStream → DataStream

Partitions elements, round-robin, to a subset of downstream operations. This is useful if you want to have pipelines where you, for example, fan out from each parallel instance of a source to a subset of several mappers to distribute load but don’t want the full rebalance that rebalance() would incur. This would require only local data transfers instead of transferring data over network, depending on other configuration values such as the number of slots of TaskManagers.

The subset of downstream operations to which the upstream operation sends elements depends on the degree of parallelism of both the upstream and downstream operation. For example, if the upstream operation has parallelism 2 and the downstream operation has parallelism 6, then one upstream operation would distribute elements to three downstream operations while the other upstream operation would distribute to the other three downstream operations. If, on the other hand, the downstream operation has parallelism 2 while the upstream operation has parallelism 6 then three upstream operations would distribute to one downstream operation while the other three upstream operations would distribute to the other downstream operation.

In cases where the different parallelisms are not multiples of each other one or several downstream operations will have a differing number of inputs from upstream operations.

Please see this figure for a visualization of the connection pattern in the above example:

Checkpoint barriers in data streams

Java

  1. dataStream.rescale();

Scala

  1. dataStream.rescale()

Python

  1. data_stream.rescale()

Broadcasting

DataStream → DataStream

Broadcasts elements to every partition.

Java

  1. dataStream.broadcast();

Scala

  1. dataStream.broadcast()

Python

  1. data_stream.broadcast()

算子链和资源组

将两个算子链接在一起能使得它们在同一个线程中执行，从而提升性能。Flink 默认会将能链接的算子尽可能地进行链接(例如， 两个 map 转换操作)。此外， Flink 还提供了对链接更细粒度控制的 API 以满足更多需求：

如果想对整个作业禁用算子链，可以调用 StreamExecutionEnvironment.disableOperatorChaining()。下列方法还提供了更细粒度的控制。需要注 意的是， 这些方法只能在 DataStream 转换操作后才能被调用，因为它们只对前一次数据转换生效。例如，可以 someStream.map(...).startNewChain() 这样调用，而不能 someStream.startNewChain()这样。

一个资源组对应着 Flink 中的一个 slot 槽，更多细节请看slots 槽。 你可以根据需要手动地将各个算子隔离到不同的 slot 中。

Start New Chain

Begin a new chain, starting with this operator. The two mappers will be chained, and filter will not be chained to the first mapper.

Java

  1. someStream.filter(...).map(...).startNewChain().map(...);

Scala

  1. someStream.filter(...).map(...).startNewChain().map(...)

Python

  1. some_stream.filter(...).map(...).start_new_chain().map(...)

Disable Chaining

Do not chain the map operator.

Java

  1. someStream.map(...).disableChaining();

Scala

  1. someStream.map(...).disableChaining()

Python

  1. some_stream.map(...).disable_chaining()

Set Slot Sharing Group

Set the slot sharing group of an operation. Flink will put operations with the same slot sharing group into the same slot while keeping operations that don’t have the slot sharing group in other slots. This can be used to isolate slots. The slot sharing group is inherited from input operations if all input operations are in the same slot sharing group. The name of the default slot sharing group is “default”, operations can explicitly be put into this group by calling slotSharingGroup(“default”).

Java

  1. someStream.filter(...).slotSharingGroup("name");

Scala

  1. someStream.filter(...).slotSharingGroup("name")

Python

  1. some_stream.filter(...).slot_sharing_group("name")