Pod template

The operator CRD is designed to have a minimal set of direct, short-hand CRD settings to express the most basic attributes of a deployment. For all other settings the CRD provides the flinkConfiguration and podTemplate fields.

Pod templates permit customization of the Flink job and task manager pods, for example to specify volume mounts, ephemeral storage, sidecar containers etc.

Pod templates can be layered, as shown in the example below. A common pod template may hold the settings that apply to both job and task manager, like volumeMounts. Another template under job or task manager can define additional settings that supplement or override those in the common template, such as a task manager sidecar.

The operator is going to merge the common and specific templates for job and task manager respectively.

Here the full example:

  1. apiVersion: flink.apache.org/v1beta1
  2. kind: FlinkDeployment
  3. metadata:
  4.   namespace: default
  5.   name: pod-template-example
  6. spec:
  7.   image: flink:1.17
  8.   flinkVersion: v1_17
  9.   flinkConfiguration:
  10.     taskmanager.numberOfTaskSlots: "2"
  11.   serviceAccount: flink
  12.   podTemplate:
  13.     apiVersion: v1
  14.     kind: Pod
  15.     metadata:
  16.       name: pod-template
  17.     spec:
  18.       containers:
  19.         # Do not change the main container name
  20.         - name: flink-main-container
  21.           volumeMounts:
  22.             - mountPath: /opt/flink/log
  23.               name: flink-logs
  24.         # Sample sidecar container
  25.         - name: fluentbit
  26.           image: fluent/fluent-bit:1.8.12-debug
  27.           command: [ 'sh','-c','/fluent-bit/bin/fluent-bit -i tail -p path=/flink-logs/*.log -p multiline.parser=java -o stdout' ]
  28.           volumeMounts:
  29.             - mountPath: /flink-logs
  30.               name: flink-logs
  31.       volumes:
  32.         - name: flink-logs
  33.           emptyDir: { }
  34.   jobManager:
  35.     resource:
  36.       memory: "2048m"
  37.       cpu: 1
  38.   taskManager:
  39.     resource:
  40.       memory: "2048m"
  41.       cpu: 1
  42.     podTemplate:
  43.       apiVersion: v1
  44.       kind: Pod
  45.       metadata:
  46.         name: task-manager-pod-template
  47.       spec:
  48.         initContainers:
  49.           # Sample sidecar container
  50.           - name: busybox
  51.             image: busybox:1.35.0
  52.             command: [ 'sh','-c','echo hello from task manager' ]
  53.   job:
  54.     jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar
  55.     parallelism: 2

When using the operator with Flink native Kubernetes integration, please refer to pod template field precedence.

Array Merging Behaviour

When layering pod templates (defining both a top level and jobmanager specific podtemplate for example) the corresponding yamls are merged together.

The default behaviour of the pod template mechanism is to merge array arrays by merging the objects in the respective array positions. This requires that containers in the podTemplates are defined in the same order otherwise the results may be undefined.

Default behaviour (merge by position):

  1. arr1: [{name: a, p1: v1}, {name: b, p1: v1}]
  2. arr1: [{name: a, p2: v2}, {name: c, p2: v2}]
  4. merged: [{name: a, p1: v1, p2: v2}, {name: c, p1: v1, p2: v2}]

The operator supports an alternative array merging mechanism that can be enabled by the kubernetes.operator.pod-template.merge-arrays-by-name flag. When true, instead of the default positional merging, object array elements that have a name property defined will be merged by their name and the resulting array will be a union of the two input arrays.

Merge by name:

  1. arr1: [{name: a, p1: v1}, {name: b, p1: v1}]
  2. arr1: [{name: a, p2: v2}, {name: c, p2: v2}]
  4. merged: [{name: a, p1: v1, p2: v2}, {name: b, p1: v1}, {name: c, p2: v2}]

Merging by name can we be very convenient when merging container specs or when the base and override templates are not defined together.