Explore Termination Behavior for Pods And Their Endpoints

Once you connected your Application with Service following steps like those outlined in Connecting Applications with Services, you have a continuously running, replicated application, that is exposed on a network. This tutorial helps you look at the termination flow for Pods and to explore ways to implement graceful connection draining.

Termination process for Pods and their endpoints

There are often cases when you need to terminate a Pod - be it for upgrade or scale down. In order to improve application availability, it may be important to implement a proper active connections draining.

This tutorial explains the flow of Pod termination in connection with the corresponding endpoint state and removal by using a simple nginx web server to demonstrate the concept.

Example flow with endpoint termination

The following is the example of the flow described in the Termination of Pods document.

Let’s say you have a Deployment containing of a single nginx replica (just for demonstration purposes) and a Service:

  1. service/pod-with-graceful-termination.yaml
  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4. name: nginx-deployment
  5. labels:
  6. app: nginx
  7. spec:
  8. replicas: 1
  9. selector:
  10. matchLabels:
  11. app: nginx
  12. template:
  13. metadata:
  14. labels:
  15. app: nginx
  16. spec:
  17. terminationGracePeriodSeconds: 120 # extra long grace period
  18. containers:
  19. - name: nginx
  20. image: nginx:latest
  21. ports:
  22. - containerPort: 80
  23. lifecycle:
  24. preStop:
  25. exec:
  26. # Real life termination may take any time up to terminationGracePeriodSeconds.
  27. # In this example - just hang around for at least the duration of terminationGracePeriodSeconds,
  28. # at 120 seconds container will be forcibly terminated.
  29. # Note, all this time nginx will keep processing requests.
  30. command: [
  31. "/bin/sh", "-c", "sleep 180"
  32. ]
  1. service/explore-graceful-termination-nginx.yaml
  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4. name: nginx-service
  5. spec:
  6. selector:
  7. app: nginx
  8. ports:
  9. - protocol: TCP
  10. port: 80
  11. targetPort: 80

Now create the Deployment Pod and Service using the above files:

  1. kubectl apply -f pod-with-graceful-termination.yaml
  2. kubectl apply -f explore-graceful-termination-nginx.yaml

Once the Pod and Service are running, you can get the name of any associated EndpointSlices:

  1. kubectl get endpointslice

The output is similar to this:

  1. NAME ADDRESSTYPE PORTS ENDPOINTS AGE
  2. nginx-service-6tjbr IPv4 80 10.12.1.199,10.12.1.201 22m

You can see its status, and validate that there is one endpoint registered:

  1. kubectl get endpointslices -o json -l kubernetes.io/service-name=nginx-service

The output is similar to this:

  1. {
  2. "addressType": "IPv4",
  3. "apiVersion": "discovery.k8s.io/v1",
  4. "endpoints": [
  5. {
  6. "addresses": [
  7. "10.12.1.201"
  8. ],
  9. "conditions": {
  10. "ready": true,
  11. "serving": true,
  12. "terminating": false

Now let’s terminate the Pod and validate that the Pod is being terminated respecting the graceful termination period configuration:

  1. kubectl delete pod nginx-deployment-7768647bf9-b4b9s

All pods:

  1. kubectl get pods

The output is similar to this:

  1. NAME READY STATUS RESTARTS AGE
  2. nginx-deployment-7768647bf9-b4b9s 1/1 Terminating 0 4m1s
  3. nginx-deployment-7768647bf9-rkxlw 1/1 Running 0 8s

You can see that the new pod got scheduled.

While the new endpoint is being created for the new Pod, the old endpoint is still around in the terminating state:

  1. kubectl get endpointslice -o json nginx-service-6tjbr

The output is similar to this:

  1. {
  2. "addressType": "IPv4",
  3. "apiVersion": "discovery.k8s.io/v1",
  4. "endpoints": [
  5. {
  6. "addresses": [
  7. "10.12.1.201"
  8. ],
  9. "conditions": {
  10. "ready": false,
  11. "serving": true,
  12. "terminating": true
  13. },
  14. "nodeName": "gke-main-default-pool-dca1511c-d17b",
  15. "targetRef": {
  16. "kind": "Pod",
  17. "name": "nginx-deployment-7768647bf9-b4b9s",
  18. "namespace": "default",
  19. "uid": "66fa831c-7eb2-407f-bd2c-f96dfe841478"
  20. },
  21. "zone": "us-central1-c"
  22. },
  23. {
  24. "addresses": [
  25. "10.12.1.202"
  26. ],
  27. "conditions": {
  28. "ready": true,
  29. "serving": true,
  30. "terminating": false
  31. },
  32. "nodeName": "gke-main-default-pool-dca1511c-d17b",
  33. "targetRef": {
  34. "kind": "Pod",
  35. "name": "nginx-deployment-7768647bf9-rkxlw",
  36. "namespace": "default",
  37. "uid": "722b1cbe-dcd7-4ed4-8928-4a4d0e2bbe35"
  38. },
  39. "zone": "us-central1-c"

This allows applications to communicate their state during termination and clients (such as load balancers) to implement a connections draining functionality. These clients may detect terminating endpoints and implement a special logic for them.

In Kubernetes, endpoints that are terminating always have their ready status set as as false. This needs to happen for backward compatibility, so existing load balancers will not use it for regular traffic. If traffic draining on terminating pod is needed, the actual readiness can be checked as a condition serving.

When Pod is deleted, the old endpoint will also be deleted.

What’s next