Testing HPAs with kubectl

This document describes how to check the status of your HPAs after scaling them up or down with your load testing tool. For information on how to check the status from the Rancher UI (at least version 2.3.x), refer to Managing HPAs with the Rancher UI.

For HPA to work correctly, service deployments should have resources request definitions for containers. Follow this hello-world example to test if HPA is working correctly.

  1. Configure kubectl to connect to your Kubernetes cluster.

  2. Copy the hello-world deployment manifest below.

    Hello World Manifest

    1. apiVersion: apps/v1beta2
    2. kind: Deployment
    3. metadata:
    4.   labels:
    5.     app: hello-world
    6.   name: hello-world
    7.   namespace: default
    8. spec:
    9.   replicas: 1
    10.   selector:
    11.     matchLabels:
    12.       app: hello-world
    13.   strategy:
    14.     rollingUpdate:
    15.       maxSurge: 1
    16.       maxUnavailable: 0
    17.     type: RollingUpdate
    18.   template:
    19.     metadata:
    20.       labels:
    21.         app: hello-world
    22.     spec:
    23.       containers:
    24.       - image: rancher/hello-world
    25.         imagePullPolicy: Always
    26.         name: hello-world
    27.         resources:
    28.           requests:
    29.             cpu: 500m
    30.             memory: 64Mi
    31.         ports:
    32.         - containerPort: 80
    33.           protocol: TCP
    34.       restartPolicy: Always
    35. ---
    36. apiVersion: v1
    37. kind: Service
    38. metadata:
    39.   name: hello-world
    40.   namespace: default
    41. spec:
    42.   ports:
    43.   - port: 80
    44.     protocol: TCP
    45.     targetPort: 80
    46.   selector:
    47.     app: hello-world

  3. Deploy it to your cluster.

    1. # kubectl create -f <HELLO_WORLD_MANIFEST>

  4. Copy one of the HPAs below based on the metric type you’re using:

    Hello World HPA: Resource Metrics

    1. apiVersion: autoscaling/v2beta1
    2. kind: HorizontalPodAutoscaler
    3. metadata:
    4.   name: hello-world
    5.   namespace: default
    6. spec:
    7.   scaleTargetRef:
    8.     apiVersion: extensions/v1beta1
    9.     kind: Deployment
    10.     name: hello-world
    11.   minReplicas: 1
    12.   maxReplicas: 10
    13.   metrics:
    14.   - type: Resource
    15.     resource:
    16.       name: cpu
    17.       targetAverageUtilization: 50
    18.   - type: Resource
    19.     resource:
    20.       name: memory
    21.       targetAverageValue: 1000Mi

    Hello World HPA: Custom Metrics

    1. apiVersion: autoscaling/v2beta1
    2. kind: HorizontalPodAutoscaler
    3. metadata:
    4.   name: hello-world
    5.   namespace: default
    6. spec:
    7.   scaleTargetRef:
    8.     apiVersion: extensions/v1beta1
    9.     kind: Deployment
    10.     name: hello-world
    11.   minReplicas: 1
    12.   maxReplicas: 10
    13.   metrics:
    14.   - type: Resource
    15.     resource:
    16.       name: cpu
    17.       targetAverageUtilization: 50
    18.   - type: Resource
    19.     resource:
    20.       name: memory
    21.       targetAverageValue: 100Mi
    22.   - type: Pods
    23.     pods:
    24.       metricName: cpu_system
    25.       targetAverageValue: 20m

  5. View the HPA info and description. Confirm that metric data is shown.

    Resource Metrics

    1. Enter the following commands.

      1. # kubectl get hpa
      2. NAME          REFERENCE                TARGETS                     MINPODS   MAXPODS   REPLICAS   AGE
      3. hello-world   Deployment/hello-world   1253376 / 100Mi, 0% / 50%   1         10        1          6m
      4. # kubectl describe hpa
      5. Name:                                                  hello-world
      6. Namespace:                                             default
      7. Labels:                                                <none>
      8. Annotations:                                           <none>
      9. CreationTimestamp:                                     Mon, 23 Jul 2018 20:21:16 +0200
      10. Reference:                                             Deployment/hello-world
      11. Metrics:                                               ( current / target )
      12.   resource memory on pods:                             1253376 / 100Mi
      13.   resource cpu on pods  (as a percentage of request):  0% (0) / 50%
      14. Min replicas:                                          1
      15. Max replicas:                                          10
      16. Conditions:
      17.   Type            Status  Reason              Message
      18.   ----            ------  ------              -------
      19.   AbleToScale     True    ReadyForNewScale    the last scale time was sufficiently old as to warrant a new scale
      20.   ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource
      21.   ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
      22. Events:           <none>

      Custom Metrics

    2. Enter the following command.

      1. # kubectl describe hpa

      You should receive the output that follows.

      1. Name:                                                  hello-world
      2. Namespace:                                             default
      3. Labels:                                                <none>
      4. Annotations:                                           <none>
      5. CreationTimestamp:                                     Tue, 24 Jul 2018 18:36:28 +0200
      6. Reference:                                             Deployment/hello-world
      7. Metrics:                                               ( current / target )
      8.   resource memory on pods:                             3514368 / 100Mi
      9.   "cpu_system" on pods:                                0 / 20m
      10.   resource cpu on pods  (as a percentage of request):  0% (0) / 50%
      11. Min replicas:                                          1
      12. Max replicas:                                          10
      13. Conditions:
      14.   Type            Status  Reason              Message
      15.   ----            ------  ------              -------
      16.   AbleToScale     True    ReadyForNewScale    the last scale time was sufficiently old as to warrant a new scale
      17.   ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource
      18.   ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
      19. Events:           <none>

  6. Generate a load for the service to test that your pods autoscale as intended. You can use any load-testing tool (Hey, Gatling, etc.), but we’re using Hey.

  7. Test that pod autoscaling works as intended.

    To Test Autoscaling Using Resource Metrics:

    Upscale to 2 Pods: CPU Usage Up to Target

    Use your load testing tool to scale up to two pods based on CPU Usage.

    1. View your HPA.

      1. # kubectl describe hpa

      You should receive output similar to what follows.

      1. Name:                                                  hello-world
      2. Namespace:                                             default
      3. Labels:                                                <none>
      4. Annotations:                                           <none>
      5. CreationTimestamp:                                     Mon, 23 Jul 2018 22:22:04 +0200
      6. Reference:                                             Deployment/hello-world
      7. Metrics:                                               ( current / target )
      8.   resource memory on pods:                             10928128 / 100Mi
      9.   resource cpu on pods  (as a percentage of request):  56% (280m) / 50%
      10. Min replicas:                                          1
      11. Max replicas:                                          10
      12. Conditions:
      13.   Type            Status  Reason              Message
      14.   ----            ------  ------              -------
      15.   AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 2
      16.   ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
      17.   ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
      18. Events:
      19.   Type    Reason             Age   From                       Message
      20.   ----    ------             ----  ----                       -------
      21.   Normal  SuccessfulRescale  13s   horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target

    2. Enter the following command to confirm you’ve scaled to two pods.

      1.   # kubectl get pods

      You should receive output similar to what follows:

      1.   NAME                                                     READY     STATUS    RESTARTS   AGE
      2.   hello-world-54764dfbf8-k8ph2                             1/1       Running   0          1m
      3.   hello-world-54764dfbf8-q6l4v                             1/1       Running   0          3h

      Upscale to 3 pods: CPU Usage Up to Target

    Use your load testing tool to upscale to 3 pods based on CPU usage with horizontal-pod-autoscaler-upscale-delay set to 3 minutes.

    1. Enter the following command.

      1. # kubectl describe hpa

      You should receive output similar to what follows

      1.   Name:                                                  hello-world
      2.   Namespace:                                             default
      3.   Labels:                                                <none>
      4.   Annotations:                                           <none>
      5.   CreationTimestamp:                                     Mon, 23 Jul 2018 22:22:04 +0200
      6.   Reference:                                             Deployment/hello-world
      7.   Metrics:                                               ( current / target )
      8.     resource memory on pods:                             9424896 / 100Mi
      9.     resource cpu on pods  (as a percentage of request):  66% (333m) / 50%
      10.   Min replicas:                                          1
      11.   Max replicas:                                          10
      12.   Conditions:
      13.     Type            Status  Reason              Message
      14.     ----            ------  ------              -------
      15.     AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 3
      16.     ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
      17.     ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
      18.   Events:
      19.     Type    Reason             Age   From                       Message
      20.     ----    ------             ----  ----                       -------
      21.     Normal  SuccessfulRescale  4m    horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
      22.     Normal  SuccessfulRescale  16s   horizontal-pod-autoscaler  New size: 3; reason: cpu resource utilization (percentage of request) above target

    2. Enter the following command to confirm three pods are running.

      1. # kubectl get pods

      You should receive output similar to what follows.

      1.   NAME                                                     READY     STATUS    RESTARTS   AGE
      2.   hello-world-54764dfbf8-f46kh                             0/1       Running   0          1m
      3.   hello-world-54764dfbf8-k8ph2                             1/1       Running   0          5m
      4.   hello-world-54764dfbf8-q6l4v                             1/1       Running   0          3h

      Downscale to 1 Pod: All Metrics Below Target

    Use your load testing to scale down to 1 pod when all metrics are below target for horizontal-pod-autoscaler-downscale-delay (5 minutes by default).

    1. Enter the following command.

      1. # kubectl describe hpa

      You should receive output similar to what follows.

      1.   Name:                                                  hello-world
      2.   Namespace:                                             default
      3.   Labels:                                                <none>
      4.   Annotations:                                           <none>
      5.   CreationTimestamp:                                     Mon, 23 Jul 2018 22:22:04 +0200
      6.   Reference:                                             Deployment/hello-world
      7.   Metrics:                                               ( current / target )
      8.     resource memory on pods:                             10070016 / 100Mi
      9.     resource cpu on pods  (as a percentage of request):  0% (0) / 50%
      10.   Min replicas:                                          1
      11.   Max replicas:                                          10
      12.   Conditions:
      13.     Type            Status  Reason              Message
      14.     ----            ------  ------              -------
      15.     AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 1
      16.     ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource
      17.     ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
      18.   Events:
      19.     Type    Reason             Age   From                       Message
      20.     ----    ------             ----  ----                       -------
      21.     Normal  SuccessfulRescale  10m   horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
      22.     Normal  SuccessfulRescale  6m    horizontal-pod-autoscaler  New size: 3; reason: cpu resource utilization (percentage of request) above target
      23.     Normal  SuccessfulRescale  1s    horizontal-pod-autoscaler  New size: 1; reason: All metrics below target
  1. **To Test Autoscaling Using Custom Metrics:**
  3.  Upscale to 2 Pods: CPU Usage Up to Target
  5. Use your load testing tool to upscale two pods based on CPU usage.
  7. 1.  Enter the following command.
  9.     ```
  10.     # kubectl describe hpa
  11.     ```
  13.     You should receive output similar to what follows.
  15.     ```
  16.     Name:                                                  hello-world
  17.     Namespace:                                             default
  18.     Labels:                                                <none>
  19.     Annotations:                                           <none>
  20.     CreationTimestamp:                                     Tue, 24 Jul 2018 18:01:11 +0200
  21.     Reference:                                             Deployment/hello-world
  22.     Metrics:                                               ( current / target )
  23.       resource memory on pods:                             8159232 / 100Mi
  24.       "cpu_system" on pods:                                7m / 20m
  25.       resource cpu on pods  (as a percentage of request):  64% (321m) / 50%
  26.     Min replicas:                                          1
  27.     Max replicas:                                          10
  28.     Conditions:
  29.       Type            Status  Reason              Message
  30.       ----            ------  ------              -------
  31.       AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 2
  32.       ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  33.       ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
  34.     Events:
  35.       Type    Reason             Age   From                       Message
  36.       ----    ------             ----  ----                       -------
  37.       Normal  SuccessfulRescale  16s   horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
  38.     ```
  40. 2.  Enter the following command to confirm two pods are running.
  42.     ```
  43.     # kubectl get pods
  44.     ```
  46.     You should receive output similar to what follows.
  48.     ```
  49.         NAME                           READY     STATUS    RESTARTS   AGE
  50.         hello-world-54764dfbf8-5pfdr   1/1       Running   0          3s
  51.         hello-world-54764dfbf8-q6l82   1/1       Running   0          6h
  52.     ```
  54.  Upscale to 3 Pods: CPU Usage Up to Target
  56. Use your load testing tool to scale up to three pods when the cpu\_system usage limit is up to target.
  58. 1.  Enter the following command.
  60.     ```
  61.     # kubectl describe hpa
  62.     ```
  64.     You should receive output similar to what follows:
  66.     ```
  67.       Name:                                                  hello-world
  68.       Namespace:                                             default
  69.       Labels:                                                <none>
  70.       Annotations:                                           <none>
  71.       CreationTimestamp:                                     Tue, 24 Jul 2018 18:01:11 +0200
  72.       Reference:                                             Deployment/hello-world
  73.       Metrics:                                               ( current / target )
  74.         resource memory on pods:                             8374272 / 100Mi
  75.         "cpu_system" on pods:                                27m / 20m
  76.         resource cpu on pods  (as a percentage of request):  71% (357m) / 50%
  77.       Min replicas:                                          1
  78.       Max replicas:                                          10
  79.       Conditions:
  80.         Type            Status  Reason              Message
  81.         ----            ------  ------              -------
  82.         AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 3
  83.         ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  84.         ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
  85.       Events:
  86.         Type    Reason             Age   From                       Message
  87.         ----    ------             ----  ----                       -------
  88.         Normal  SuccessfulRescale  3m    horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
  89.         Normal  SuccessfulRescale  3s    horizontal-pod-autoscaler  New size: 3; reason: pods metric cpu_system above target
  90.     ```
  92. 2.  Enter the following command to confirm three pods are running.
  94.     ```
  95.     # kubectl get pods
  96.     ```
  98.     You should receive output similar to what follows:
  100.     ```
  101.       # kubectl get pods
  102.       NAME                           READY     STATUS    RESTARTS   AGE
  103.       hello-world-54764dfbf8-5pfdr   1/1       Running   0          3m
  104.       hello-world-54764dfbf8-m2hrl   1/1       Running   0          1s
  105.       hello-world-54764dfbf8-q6l82   1/1       Running   0          6h
  106.     ```
  108.  Upscale to 4 Pods: CPU Usage Up to Target
  110. Use your load testing tool to upscale to four pods based on CPU usage. `horizontal-pod-autoscaler-upscale-delay` is set to three minutes by default.
  112. 1.  Enter the following command.
  114.     ```
  115.     # kubectl describe hpa
  116.     ```
  118.     You should receive output similar to what follows.
  120.     ```
  121.       Name:                                                  hello-world
  122.       Namespace:                                             default
  123.       Labels:                                                <none>
  124.       Annotations:                                           <none>
  125.       CreationTimestamp:                                     Tue, 24 Jul 2018 18:01:11 +0200
  126.       Reference:                                             Deployment/hello-world
  127.       Metrics:                                               ( current / target )
  128.         resource memory on pods:                             8374272 / 100Mi
  129.         "cpu_system" on pods:                                27m / 20m
  130.         resource cpu on pods  (as a percentage of request):  71% (357m) / 50%
  131.       Min replicas:                                          1
  132.       Max replicas:                                          10
  133.       Conditions:
  134.         Type            Status  Reason              Message
  135.         ----            ------  ------              -------
  136.         AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 3
  137.         ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  138.         ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
  139.       Events:
  140.         Type    Reason             Age   From                       Message
  141.         ----    ------             ----  ----                       -------
  142.         Normal  SuccessfulRescale  5m    horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
  143.         Normal  SuccessfulRescale  3m    horizontal-pod-autoscaler  New size: 3; reason: pods metric cpu_system above target
  144.         Normal  SuccessfulRescale  4s    horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
  145.     ```
  147. 2.  Enter the following command to confirm four pods are running.
  149.     ```
  150.     # kubectl get pods
  151.     ```
  153.     You should receive output similar to what follows.
  155.     ```
  156.       NAME                           READY     STATUS    RESTARTS   AGE
  157.       hello-world-54764dfbf8-2p9xb   1/1       Running   0          5m
  158.       hello-world-54764dfbf8-5pfdr   1/1       Running   0          2m
  159.       hello-world-54764dfbf8-m2hrl   1/1       Running   0          1s
  160.       hello-world-54764dfbf8-q6l82   1/1       Running   0          6h
  161.     ```
  163.  Downscale to 1 Pod: All Metrics Below Target
  165. Use your load testing tool to scale down to one pod when all metrics below target for `horizontal-pod-autoscaler-downscale-delay`.
  167. 1.  Enter the following command.
  169.     ```
  170.     # kubectl describe hpa
  171.     ```
  173.     You should receive similar output to what follows.
  175.     ```
  176.         Name:                                                  hello-world
  177.         Namespace:                                             default
  178.         Labels:                                                <none>
  179.         Annotations:                                           <none>
  180.         CreationTimestamp:                                     Tue, 24 Jul 2018 18:01:11 +0200
  181.         Reference:                                             Deployment/hello-world
  182.         Metrics:                                               ( current / target )
  183.           resource memory on pods:                             8101888 / 100Mi
  184.           "cpu_system" on pods:                                8m / 20m
  185.           resource cpu on pods  (as a percentage of request):  0% (0) / 50%
  186.         Min replicas:                                          1
  187.         Max replicas:                                          10
  188.         Conditions:
  189.           Type            Status  Reason              Message
  190.           ----            ------  ------              -------
  191.           AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 1
  192.           ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource
  193.           ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
  194.         Events:
  195.           Type    Reason             Age   From                       Message
  196.           ----    ------             ----  ----                       -------
  197.           Normal  SuccessfulRescale  10m    horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
  198.           Normal  SuccessfulRescale  8m    horizontal-pod-autoscaler  New size: 3; reason: pods metric cpu_system above target
  199.           Normal  SuccessfulRescale  5m    horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
  200.           Normal   SuccessfulRescale             13s               horizontal-pod-autoscaler  New size: 1; reason: All metrics below target
  201.     ```
  203. 2.  Enter the following command to confirm a single pods is running.
  205.     ```
  206.         # kubectl get pods
  207.     ```
  209.     You should receive output similar to what follows.
  211.     ```
  212.         NAME                           READY     STATUS    RESTARTS   AGE
  213.         hello-world-54764dfbf8-q6l82   1/1       Running   0          6h
  214.     ```