Job with Pod-to-Pod Communication

In this example, you will run a Job in Indexed completion mode configured such that the pods created by the Job can communicate with each other using pod hostnames rather than pod IP addresses.

Pods within a Job might need to communicate among themselves. The user workload running in each pod could query the Kubernetes API server to learn the IPs of the other Pods, but it’s much simpler to rely on Kubernetes’ built-in DNS resolution.

Jobs in Indexed completion mode automatically set the pods’ hostname to be in the format of ${jobName}-${completionIndex}. You can use this format to deterministically build pod hostnames and enable pod communication without needing to create a client connection to the Kubernetes control plane to obtain pod hostnames/IPs via API requests.

This configuration is useful for use cases where pod networking is required but you don’t want to depend on a network connection with the Kubernetes API server.

Before you begin

You should already be familiar with the basic use of Job.

You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

Your Kubernetes server must be at or later than version v1.21. To check the version, enter kubectl version.

Note: If you are using MiniKube or a similar tool, you may need to take extra steps to ensure you have DNS.

Starting a Job with Pod-to-Pod Communication

To enable pod-to-pod communication using pod hostnames in a Job, you must do the following:

  1. Set up a headless Service with a valid label selector for the pods created by your Job. The headless service must be in the same namespace as the Job. One easy way to do this is to use the job-name: <your-job-name> selector, since the job-name label will be automatically added by Kubernetes. This configuration will trigger the DNS system to create records of the hostnames of the pods running your Job.

  2. Configure the headless service as subdomain service for the Job pods by including the following value in your Job template spec:

    1. subdomain: <headless-svc-name>

Example

Below is a working example of a Job with pod-to-pod communication via pod hostnames enabled. The Job is completed only after all pods successfully ping each other using hostnames.

Note: In the Bash script executed on each pod in the example below, the pod hostnames can be prefixed by the namespace as well if the pod needs to be reached from outside the namespace.

  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5.   name: headless-svc
  6. spec:
  7.   clusterIP: None # clusterIP must be None to create a headless service
  8.   selector:
  9.     job-name: example-job # must match Job name
  10. ---
  11. apiVersion: batch/v1
  12. kind: Job
  13. metadata:
  14.   name: example-job
  15. spec:
  16.   completions: 3
  17.   parallelism: 3
  18.   completionMode: Indexed
  19.   template:
  20.     spec:
  21.       subdomain: headless-svc # has to match Service name
  22.       restartPolicy: Never
  23.       containers:
  24.       - name: example-workload
  25.         image: bash:latest
  26.         command:
  27.         - bash
  28.         - -c
  29.         - |
  30.           for i in 0 1 2
  31.           do
  32.             gotStatus="-1"
  33.             wantStatus="0"             
  34.             while [ $gotStatus -ne $wantStatus ]
  35.             do                                       
  36.               ping -c 1 example-job-${i}.headless-svc > /dev/null 2>&1
  37.               gotStatus=$?                
  38.               if [ $gotStatus -ne $wantStatus ]; then
  39.                 echo "Failed to ping pod example-job-${i}.headless-svc, retrying in 1 second..."
  40.                 sleep 1
  41.               fi
  42.             done                                                         
  43.             echo "Successfully pinged pod: example-job-${i}.headless-svc"
  44.           done

After applying the example above, reach each other over the network using: <pod-hostname>.<headless-service-name>. You should see output similar to the following:

  1. kubectl logs example-job-0-qws42
  1. Failed to ping pod example-job-0.headless-svc, retrying in 1 second...
  2. Successfully pinged pod: example-job-0.headless-svc
  3. Successfully pinged pod: example-job-1.headless-svc
  4. Successfully pinged pod: example-job-2.headless-svc

Note: Keep in mind that the <pod-hostname>.<headless-service-name> name format used in this example would not work with DNS policy set to None or Default. You can learn more about pod DNS policies here.