Pod template

The operator CRD is designed to have a minimal set of direct, short-hand CRD settings to express the most basic attributes of a deployment. For all other settings the CRD provides the flinkConfiguration and podTemplate fields.

Pod templates permit customization of the Flink job and task manager pods, for example to specify volume mounts, ephemeral storage, sidecar containers etc.

Pod templates can be layered, as shown in the example below. A common pod template may hold the settings that apply to both job and task manager, like volumeMounts. Another template under job or task manager can define additional settings that supplement or override those in the common template, such as a task manager sidecar.

The operator is going to merge the common and specific templates for job and task manager respectively.

Here the full example:

  1. apiVersion: flink.apache.org/v1beta1
  2. kind: FlinkDeployment
  3. metadata:
  4.   namespace: default
  5.   name: pod-template-example
  6. spec:
  7.   image: flink:1.17
  8.   flinkVersion: v1_17
  9.   flinkConfiguration:
  10.     taskmanager.numberOfTaskSlots: "2"
  11.   serviceAccount: flink
  12.   podTemplate:
  13.     spec:
  14.       containers:
  15.         # Do not change the main container name
  16.         - name: flink-main-container
  17.           volumeMounts:
  18.             - mountPath: /opt/flink/log
  19.               name: flink-logs
  20.         # Sample sidecar container
  21.         - name: fluentbit
  22.           image: fluent/fluent-bit:1.8.12-debug
  23.           command: [ 'sh','-c','/fluent-bit/bin/fluent-bit -i tail -p path=/flink-logs/*.log -p multiline.parser=java -o stdout' ]
  24.           volumeMounts:
  25.             - mountPath: /flink-logs
  26.               name: flink-logs
  27.       volumes:
  28.         - name: flink-logs
  29.           emptyDir: { }
  30.   jobManager:
  31.     resource:
  32.       memory: "2048m"
  33.       cpu: 1
  34.   taskManager:
  35.     resource:
  36.       memory: "2048m"
  37.       cpu: 1
  38.     podTemplate:
  39.       spec:
  40.         initContainers:
  41.           # Sample sidecar container
  42.           - name: busybox
  43.             image: busybox:1.35.0
  44.             command: [ 'sh','-c','echo hello from task manager' ]
  45.   job:
  46.     jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar
  47.     parallelism: 2

When using the operator with Flink native Kubernetes integration, please refer to pod template field precedence.

Array Merging Behaviour

When layering pod templates (defining both a top level and jobmanager specific podtemplate for example) the corresponding yamls are merged together.

The default behaviour of the pod template mechanism is to merge array arrays by merging the objects in the respective array positions. This requires that containers in the podTemplates are defined in the same order otherwise the results may be undefined.

Default behaviour (merge by position):

  1. arr1: [{name: a, p1: v1}, {name: b, p1: v1}]
  2. arr1: [{name: a, p2: v2}, {name: c, p2: v2}]
  4. merged: [{name: a, p1: v1, p2: v2}, {name: c, p1: v1, p2: v2}]

The operator supports an alternative array merging mechanism that can be enabled by the kubernetes.operator.pod-template.merge-arrays-by-name flag. When true, instead of the default positional merging, object array elements that have a name property defined will be merged by their name and the resulting array will be a union of the two input arrays.

Merge by name:

  1. arr1: [{name: a, p1: v1}, {name: b, p1: v1}]
  2. arr1: [{name: a, p2: v2}, {name: c, p2: v2}]
  4. merged: [{name: a, p1: v1, p2: v2}, {name: b, p1: v1}, {name: c, p2: v2}]

Merging by name can we be very convenient when merging container specs or when the base and override templates are not defined together.