Fine Parallel Processing Using a Work Queue

In this example, you will run a Kubernetes Job that runs multiple parallel tasks as worker processes, each running as a separate Pod.

In this example, as each pod is created, it picks up one unit of work from a task queue, processes it, and repeats until the end of the queue is reached.

Here is an overview of the steps in this example:

  1. Start a storage service to hold the work queue. In this example, you will use Redis to store work items. In the previous example, you used RabbitMQ. In this example, you will use Redis and a custom work-queue client library; this is because AMQP does not provide a good way for clients to detect when a finite-length work queue is empty. In practice you would set up a store such as Redis once and reuse it for the work queues of many jobs, and other things.
  2. Create a queue, and fill it with messages. Each message represents one task to be done. In this example, a message is an integer that we will do a lengthy computation on.
  3. Start a Job that works on tasks from the queue. The Job starts several pods. Each pod takes one task from the message queue, processes it, and repeats until the end of the queue is reached.

Before you begin

You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

You will need a container image registry where you can upload images to run in your cluster. The example uses Docker Hub, but you could adapt it to a different container image registry.

This task example also assumes that you have Docker installed locally. You use Docker to build container images.

Be familiar with the basic, non-parallel, use of Job.

Starting Redis

For this example, for simplicity, you will start a single instance of Redis. See the Redis Example for an example of deploying Redis scalably and redundantly.

You could also download the following files directly:

To start a single instance of Redis, you need to create the redis pod and redis service:

  1. kubectl apply -f https://k8s.io/examples/application/job/redis/redis-pod.yaml
  2. kubectl apply -f https://k8s.io/examples/application/job/redis/redis-service.yaml

Filling the queue with tasks

Now let’s fill the queue with some “tasks”. In this example, the tasks are strings to be printed.

Start a temporary interactive pod for running the Redis CLI.

  1. kubectl run -i --tty temp --image redis --command "/bin/sh"
  1. Waiting for pod default/redis2-c7h78 to be running, status is Pending, pod ready: false
  2. Hit enter for command prompt

Now hit enter, start the Redis CLI, and create a list with some work items in it.

  1. redis-cli -h redis
  1. redis:6379> rpush job2 "apple"
  2. (integer) 1
  3. redis:6379> rpush job2 "banana"
  4. (integer) 2
  5. redis:6379> rpush job2 "cherry"
  6. (integer) 3
  7. redis:6379> rpush job2 "date"
  8. (integer) 4
  9. redis:6379> rpush job2 "fig"
  10. (integer) 5
  11. redis:6379> rpush job2 "grape"
  12. (integer) 6
  13. redis:6379> rpush job2 "lemon"
  14. (integer) 7
  15. redis:6379> rpush job2 "melon"
  16. (integer) 8
  17. redis:6379> rpush job2 "orange"
  18. (integer) 9
  19. redis:6379> lrange job2 0 -1
  20. 1) "apple"
  21. 2) "banana"
  22. 3) "cherry"
  23. 4) "date"
  24. 5) "fig"
  25. 6) "grape"
  26. 7) "lemon"
  27. 8) "melon"
  28. 9) "orange"

So, the list with key job2 will be the work queue.

Note: if you do not have Kube DNS setup correctly, you may need to change the first step of the above block to redis-cli -h $REDIS_SERVICE_HOST.

Create a container image

Now you are ready to create an image that will process the work in that queue.

You’re going to use a Python worker program with a Redis client to read the messages from the message queue.

A simple Redis work queue client library is provided, called rediswq.py (Download).

The “worker” program in each Pod of the Job uses the work queue client library to get work. Here it is:

  1. application/job/redis/worker.py
  1. #!/usr/bin/env python
  3. import time
  4. import rediswq
  6. host="redis"
  7. # Uncomment next two lines if you do not have Kube-DNS working.
  8. # import os
  9. # host = os.getenv("REDIS_SERVICE_HOST")
  11. q = rediswq.RedisWQ(name="job2", host=host)
  12. print("Worker with sessionID: " +  q.sessionID())
  13. print("Initial queue state: empty=" + str(q.empty()))
  14. while not q.empty():
  15.   item = q.lease(lease_secs=10, block=True, timeout=2) 
  16.   if item is not None:
  17.     itemstr = item.decode("utf-8")
  18.     print("Working on " + itemstr)
  19.     time.sleep(10) # Put your actual work here instead of sleep.
  20.     q.complete(item)
  21.   else:
  22.     print("Waiting for work")
  23. print("Queue empty, exiting")

You could also download worker.py, rediswq.py, and Dockerfile files, then build the container image. Here’s an example using Docker to do the image build:

  1. docker build -t job-wq-2 .

Push the image

For the Docker Hub, tag your app image with your username and push to the Hub with the below commands. Replace <username> with your Hub username.

  1. docker tag job-wq-2 <username>/job-wq-2
  2. docker push <username>/job-wq-2

You need to push to a public repository or configure your cluster to be able to access your private repository.

Defining a Job

Here is a manifest for the Job you will create:

  1. application/job/redis/job.yaml
  1. apiVersion: batch/v1
  2. kind: Job
  3. metadata:
  4.   name: job-wq-2
  5. spec:
  6.   parallelism: 2
  7.   template:
  8.     metadata:
  9.       name: job-wq-2
  10.     spec:
  11.       containers:
  12.       - name: c
  13.         image: gcr.io/myproject/job-wq-2
  14.       restartPolicy: OnFailure

Note:

Be sure to edit the manifest to change gcr.io/myproject to your own path.

In this example, each pod works on several items from the queue and then exits when there are no more items. Since the workers themselves detect when the workqueue is empty, and the Job controller does not know about the workqueue, it relies on the workers to signal when they are done working. The workers signal that the queue is empty by exiting with success. So, as soon as any worker exits with success, the controller knows the work is done, and that the Pods will exit soon. So, you need to leave the completion count of the Job unset. The job controller will wait for the other pods to complete too.

Running the Job

So, now run the Job:

  1. # this assumes you downloaded and then edited the manifest already
  2. kubectl apply -f ./job.yaml

Now wait a bit, then check on the Job:

  1. kubectl describe jobs/job-wq-2
  1. Name:             job-wq-2
  2. Namespace:        default
  3. Selector:         controller-uid=b1c7e4e3-92e1-11e7-b85e-fa163ee3c11f
  4. Labels:           controller-uid=b1c7e4e3-92e1-11e7-b85e-fa163ee3c11f
  5.                   job-name=job-wq-2
  6. Annotations:      <none>
  7. Parallelism:      2
  8. Completions:      <unset>
  9. Start Time:       Mon, 11 Jan 2022 17:07:59 +0000
  10. Pods Statuses:    1 Running / 0 Succeeded / 0 Failed
  11. Pod Template:
  12.   Labels:       controller-uid=b1c7e4e3-92e1-11e7-b85e-fa163ee3c11f
  13.                 job-name=job-wq-2
  14.   Containers:
  15.    c:
  16.     Image:              container-registry.example/exampleproject/job-wq-2
  17.     Port:
  18.     Environment:        <none>
  19.     Mounts:             <none>
  20.   Volumes:              <none>
  21. Events:
  22.   FirstSeen    LastSeen    Count    From            SubobjectPath    Type        Reason            Message
  23.   ---------    --------    -----    ----            -------------    --------    ------            -------
  24.   33s          33s         1        {job-controller }                Normal      SuccessfulCreate  Created pod: job-wq-2-lglf8

You can wait for the Job to succeed, with a timeout:

  1. # The check for condition name is case insensitive
  2. kubectl wait --for=condition=complete --timeout=300s job/job-wq-2
  1. kubectl logs pods/job-wq-2-7r7b2
  1. Worker with sessionID: bbd72d0a-9e5c-4dd6-abf6-416cc267991f
  2. Initial queue state: empty=False
  3. Working on banana
  4. Working on date
  5. Working on lemon

As you can see, one of the pods for this Job worked on several work units.

Alternatives

If running a queue service or modifying your containers to use a work queue is inconvenient, you may want to consider one of the other job patterns.

If you have a continuous stream of background processing work to run, then consider running your background workers with a ReplicaSet instead, and consider running a background processing library such as https://github.com/resque/resque.