使用 kubectl 测试 HPA

本文介绍如何在使用负载测试工具扩缩 HPA 后检查 HPA 的状态。有关使用 Rancher UI(最低版本 2.3.x)检查状态的信息,请参阅使用 Rancher UI 管理 HPA

要让 HPA 正常工作,服务部署应该具有容器的资源请求定义。按照以下 hello-world 示例测试 HPA 是否正常工作。

  1. kubectl 连接到你的 Kubernetes 集群。

  2. 复制下方的 hello-world 部署清单。

    Hello World 清单

    1. apiVersion: apps/v1beta2
    2. kind: Deployment
    3. metadata:
    4. labels:
    5. app: hello-world
    6. name: hello-world
    7. namespace: default
    8. spec:
    9. replicas: 1
    10. selector:
    11. matchLabels:
    12. app: hello-world
    13. strategy:
    14. rollingUpdate:
    15. maxSurge: 1
    16. maxUnavailable: 0
    17. type: RollingUpdate
    18. template:
    19. metadata:
    20. labels:
    21. app: hello-world
    22. spec:
    23. containers:
    24. - image: rancher/hello-world
    25. imagePullPolicy: Always
    26. name: hello-world
    27. resources:
    28. requests:
    29. cpu: 500m
    30. memory: 64Mi
    31. ports:
    32. - containerPort: 80
    33. protocol: TCP
    34. restartPolicy: Always
    35. ---
    36. apiVersion: v1
    37. kind: Service
    38. metadata:
    39. name: hello-world
    40. namespace: default
    41. spec:
    42. ports:
    43. - port: 80
    44. protocol: TCP
    45. targetPort: 80
    46. selector:
    47. app: hello-world
  3. 将其部署到你的集群。

    1. # kubectl create -f <HELLO_WORLD_MANIFEST>
  4. 根据你使用的指标类型复制以下其中一个 HPA:

    Hello World HPA:资源指标

    1. apiVersion: autoscaling/v2beta1
    2. kind: HorizontalPodAutoscaler
    3. metadata:
    4. name: hello-world
    5. namespace: default
    6. spec:
    7. scaleTargetRef:
    8. apiVersion: extensions/v1beta1
    9. kind: Deployment
    10. name: hello-world
    11. minReplicas: 1
    12. maxReplicas: 10
    13. metrics:
    14. - type: Resource
    15. resource:
    16. name: cpu
    17. targetAverageUtilization: 50
    18. - type: Resource
    19. resource:
    20. name: memory
    21. targetAverageValue: 1000Mi

    Hello World HPA:自定义指标

    1. apiVersion: autoscaling/v2beta1
    2. kind: HorizontalPodAutoscaler
    3. metadata:
    4. name: hello-world
    5. namespace: default
    6. spec:
    7. scaleTargetRef:
    8. apiVersion: extensions/v1beta1
    9. kind: Deployment
    10. name: hello-world
    11. minReplicas: 1
    12. maxReplicas: 10
    13. metrics:
    14. - type: Resource
    15. resource:
    16. name: cpu
    17. targetAverageUtilization: 50
    18. - type: Resource
    19. resource:
    20. name: memory
    21. targetAverageValue: 100Mi
    22. - type: Pods
    23. pods:
    24. metricName: cpu_system
    25. targetAverageValue: 20m
  5. 查看 HPA 信息和描述。确认显示的指标数据。

    资源指标

    1. 输入以下命令:

      1. # kubectl get hpa
      2. NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
      3. hello-world Deployment/hello-world 1253376 / 100Mi, 0% / 50% 1 10 1 6m
      4. # kubectl describe hpa
      5. Name: hello-world
      6. Namespace: default
      7. Labels: <none>
      8. Annotations: <none>
      9. CreationTimestamp: Mon, 23 Jul 2018 20:21:16 +0200
      10. Reference: Deployment/hello-world
      11. Metrics: ( current / target )
      12. resource memory on pods: 1253376 / 100Mi
      13. resource cpu on pods (as a percentage of request): 0% (0) / 50%
      14. Min replicas: 1
      15. Max replicas: 10
      16. Conditions:
      17. Type Status Reason Message
      18. ---- ------ ------ -------
      19. AbleToScale True ReadyForNewScale the last scale time was sufficiently old as to warrant a new scale
      20. ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from memory resource
      21. ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
      22. Events: <none>

      自定义指标

    2. 输入以下命令:

      1. # kubectl describe hpa

      你应该会看到类似以下的输出:

      1. Name: hello-world
      2. Namespace: default
      3. Labels: <none>
      4. Annotations: <none>
      5. CreationTimestamp: Tue, 24 Jul 2018 18:36:28 +0200
      6. Reference: Deployment/hello-world
      7. Metrics: ( current / target )
      8. resource memory on pods: 3514368 / 100Mi
      9. "cpu_system" on pods: 0 / 20m
      10. resource cpu on pods (as a percentage of request): 0% (0) / 50%
      11. Min replicas: 1
      12. Max replicas: 10
      13. Conditions:
      14. Type Status Reason Message
      15. ---- ------ ------ -------
      16. AbleToScale True ReadyForNewScale the last scale time was sufficiently old as to warrant a new scale
      17. ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from memory resource
      18. ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
      19. Events: <none>
  6. 为服务生成负载,从而测试你的 pod 是否按预期进行了自动扩缩。你可以使用任何负载测试工具(Hey、Gatling 等),我们使用的是 Hey

  7. 测试 pod 自动扩缩是否按预期工作。

    使用资源指标测试自动扩缩:

    扩展到两个 Pod:CPU 用量达到目标

    使用你的负载测试工具根据 CPU 使用情况扩展到两个 Pod。

    1. 查看你的 HPA。

      1. # kubectl describe hpa

      你应该会看到类似以下的输出:

      1. Name: hello-world
      2. Namespace: default
      3. Labels: <none>
      4. Annotations: <none>
      5. CreationTimestamp: Mon, 23 Jul 2018 22:22:04 +0200
      6. Reference: Deployment/hello-world
      7. Metrics: ( current / target )
      8. resource memory on pods: 10928128 / 100Mi
      9. resource cpu on pods (as a percentage of request): 56% (280m) / 50%
      10. Min replicas: 1
      11. Max replicas: 10
      12. Conditions:
      13. Type Status Reason Message
      14. ---- ------ ------ -------
      15. AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 2
      16. ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
      17. ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
      18. Events:
      19. Type Reason Age From Message
      20. ---- ------ ---- ---- -------
      21. Normal SuccessfulRescale 13s horizontal-pod-autoscaler New size: 2; reason: cpu resource utilization (percentage of request) above target
    2. 输入以下命令,确认你已扩展到两个 pod:

      1. # kubectl get pods

      你应该会看到类似以下的输出:

      1. NAME READY STATUS RESTARTS AGE
      2. hello-world-54764dfbf8-k8ph2 1/1 Running 0 1m
      3. hello-world-54764dfbf8-q6l4v 1/1 Running 0 3h

      扩展到三个 Pod:CPU 用量达到目标

    使用你的负载测试工具根据 CPU 用量扩展到三个 Pod,并将 horizo​​ntal-pod-autoscaler-upscale-delay 设置为 3 分钟。

    1. 输入以下命令:

      1. # kubectl describe hpa

      你应该会看到类似以下的输出:

      1. Name: hello-world
      2. Namespace: default
      3. Labels: <none>
      4. Annotations: <none>
      5. CreationTimestamp: Mon, 23 Jul 2018 22:22:04 +0200
      6. Reference: Deployment/hello-world
      7. Metrics: ( current / target )
      8. resource memory on pods: 9424896 / 100Mi
      9. resource cpu on pods (as a percentage of request): 66% (333m) / 50%
      10. Min replicas: 1
      11. Max replicas: 10
      12. Conditions:
      13. Type Status Reason Message
      14. ---- ------ ------ -------
      15. AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 3
      16. ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
      17. ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
      18. Events:
      19. Type Reason Age From Message
      20. ---- ------ ---- ---- -------
      21. Normal SuccessfulRescale 4m horizontal-pod-autoscaler New size: 2; reason: cpu resource utilization (percentage of request) above target
      22. Normal SuccessfulRescale 16s horizontal-pod-autoscaler New size: 3; reason: cpu resource utilization (percentage of request) above target
    2. 输入以下命令,确认三个 pod 正在运行:

      1. # kubectl get pods

      你应该会看到类似以下的输出:

      1. NAME READY STATUS RESTARTS AGE
      2. hello-world-54764dfbf8-f46kh 0/1 Running 0 1m
      3. hello-world-54764dfbf8-k8ph2 1/1 Running 0 5m
      4. hello-world-54764dfbf8-q6l4v 1/1 Running 0 3h

      缩减到 1 个 Pod:所有指标均低于目标

    horizontal-pod-autoscaler-downscale-delay 的所有指标均低于目标(默认为 5 分钟)时,使用你的负载测试工具缩减到 1 个 pod。

    1. 输入以下命令:

      1. # kubectl describe hpa

      你应该会看到类似以下的输出:

      1. Name: hello-world
      2. Namespace: default
      3. Labels: <none>
      4. Annotations: <none>
      5. CreationTimestamp: Mon, 23 Jul 2018 22:22:04 +0200
      6. Reference: Deployment/hello-world
      7. Metrics: ( current / target )
      8. resource memory on pods: 10070016 / 100Mi
      9. resource cpu on pods (as a percentage of request): 0% (0) / 50%
      10. Min replicas: 1
      11. Max replicas: 10
      12. Conditions:
      13. Type Status Reason Message
      14. ---- ------ ------ -------
      15. AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 1
      16. ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from memory resource
      17. ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
      18. Events:
      19. Type Reason Age From Message
      20. ---- ------ ---- ---- -------
      21. Normal SuccessfulRescale 10m horizontal-pod-autoscaler New size: 2; reason: cpu resource utilization (percentage of request) above target
      22. Normal SuccessfulRescale 6m horizontal-pod-autoscaler New size: 3; reason: cpu resource utilization (percentage of request) above target
      23. Normal SuccessfulRescale 1s horizontal-pod-autoscaler New size: 1; reason: All metrics below target

    使用自定义指标测试自动缩放:

    扩展到两个 Pod:CPU 用量达到目标

    使用负载测试工具根据 CPU 用量扩展到两个 Pod。

    1. 输入以下命令:

      1. # kubectl describe hpa

      你应该会看到类似以下的输出:

      1. Name: hello-world
      2. Namespace: default
      3. Labels: <none>
      4. Annotations: <none>
      5. CreationTimestamp: Tue, 24 Jul 2018 18:01:11 +0200
      6. Reference: Deployment/hello-world
      7. Metrics: ( current / target )
      8. resource memory on pods: 8159232 / 100Mi
      9. "cpu_system" on pods: 7m / 20m
      10. resource cpu on pods (as a percentage of request): 64% (321m) / 50%
      11. Min replicas: 1
      12. Max replicas: 10
      13. Conditions:
      14. Type Status Reason Message
      15. ---- ------ ------ -------
      16. AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 2
      17. ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
      18. ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
      19. Events:
      20. Type Reason Age From Message
      21. ---- ------ ---- ---- -------
      22. Normal SuccessfulRescale 16s horizontal-pod-autoscaler New size: 2; reason: cpu resource utilization (percentage of request) above target
    2. 输入以下命令,确认两个 pod 正在运行:

      1. # kubectl get pods

      你应该会看到类似以下的输出:

      1. NAME READY STATUS RESTARTS AGE
      2. hello-world-54764dfbf8-5pfdr 1/1 Running 0 3s
      3. hello-world-54764dfbf8-q6l82 1/1 Running 0 6h

      扩展到三个 Pod:CPU 用量达到目标

    当 cpu_system 用量达到目标时,使用你的负载测试工具扩展到三个 Pod。

    1. 输入以下命令:

      1. # kubectl describe hpa

      你应该会看到类似以下的输出:

      1. Name: hello-world
      2. Namespace: default
      3. Labels: <none>
      4. Annotations: <none>
      5. CreationTimestamp: Tue, 24 Jul 2018 18:01:11 +0200
      6. Reference: Deployment/hello-world
      7. Metrics: ( current / target )
      8. resource memory on pods: 8374272 / 100Mi
      9. "cpu_system" on pods: 27m / 20m
      10. resource cpu on pods (as a percentage of request): 71% (357m) / 50%
      11. Min replicas: 1
      12. Max replicas: 10
      13. Conditions:
      14. Type Status Reason Message
      15. ---- ------ ------ -------
      16. AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 3
      17. ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
      18. ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
      19. Events:
      20. Type Reason Age From Message
      21. ---- ------ ---- ---- -------
      22. Normal SuccessfulRescale 3m horizontal-pod-autoscaler New size: 2; reason: cpu resource utilization (percentage of request) above target
      23. Normal SuccessfulRescale 3s horizontal-pod-autoscaler New size: 3; reason: pods metric cpu_system above target
    2. 输入以下命令,确认三个 pod 正在运行:

      1. # kubectl get pods

      你应该会看到类似以下的输出:

      1. # kubectl get pods
      2. NAME READY STATUS RESTARTS AGE
      3. hello-world-54764dfbf8-5pfdr 1/1 Running 0 3m
      4. hello-world-54764dfbf8-m2hrl 1/1 Running 0 1s
      5. hello-world-54764dfbf8-q6l82 1/1 Running 0 6h

      扩展到四个 Pod:CPU 用量达到目标

    使用负载测试工具根据 CPU 用量扩展到四个 Pod。horizontal-pod-autoscaler-upscale-delay 默认设置为 3 分钟。

    1. 输入以下命令:

      1. # kubectl describe hpa

      你应该会看到类似以下的输出:

      1. Name: hello-world
      2. Namespace: default
      3. Labels: <none>
      4. Annotations: <none>
      5. CreationTimestamp: Tue, 24 Jul 2018 18:01:11 +0200
      6. Reference: Deployment/hello-world
      7. Metrics: ( current / target )
      8. resource memory on pods: 8374272 / 100Mi
      9. "cpu_system" on pods: 27m / 20m
      10. resource cpu on pods (as a percentage of request): 71% (357m) / 50%
      11. Min replicas: 1
      12. Max replicas: 10
      13. Conditions:
      14. Type Status Reason Message
      15. ---- ------ ------ -------
      16. AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 3
      17. ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
      18. ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
      19. Events:
      20. Type Reason Age From Message
      21. ---- ------ ---- ---- -------
      22. Normal SuccessfulRescale 5m horizontal-pod-autoscaler New size: 2; reason: cpu resource utilization (percentage of request) above target
      23. Normal SuccessfulRescale 3m horizontal-pod-autoscaler New size: 3; reason: pods metric cpu_system above target
      24. Normal SuccessfulRescale 4s horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target
    2. 输入以下命令,确认四个 pod 正在运行:

      1. # kubectl get pods

      你应该会看到类似以下的输出:

      1. NAME READY STATUS RESTARTS AGE
      2. hello-world-54764dfbf8-2p9xb 1/1 Running 0 5m
      3. hello-world-54764dfbf8-5pfdr 1/1 Running 0 2m
      4. hello-world-54764dfbf8-m2hrl 1/1 Running 0 1s
      5. hello-world-54764dfbf8-q6l82 1/1 Running 0 6h

      缩减到 1 个 Pod:所有指标均低于目标

    horizontal-pod-autoscaler-downscale-delay 的所有指标均低于目标时,使用你的负载测试工具缩减到 1 个 pod。

    1. 输入以下命令:

      1. # kubectl describe hpa

      你应该会看到类似以下的输出:

      1. Name: hello-world
      2. Namespace: default
      3. Labels: <none>
      4. Annotations: <none>
      5. CreationTimestamp: Tue, 24 Jul 2018 18:01:11 +0200
      6. Reference: Deployment/hello-world
      7. Metrics: ( current / target )
      8. resource memory on pods: 8101888 / 100Mi
      9. "cpu_system" on pods: 8m / 20m
      10. resource cpu on pods (as a percentage of request): 0% (0) / 50%
      11. Min replicas: 1
      12. Max replicas: 10
      13. Conditions:
      14. Type Status Reason Message
      15. ---- ------ ------ -------
      16. AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 1
      17. ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from memory resource
      18. ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
      19. Events:
      20. Type Reason Age From Message
      21. ---- ------ ---- ---- -------
      22. Normal SuccessfulRescale 10m horizontal-pod-autoscaler New size: 2; reason: cpu resource utilization (percentage of request) above target
      23. Normal SuccessfulRescale 8m horizontal-pod-autoscaler New size: 3; reason: pods metric cpu_system above target
      24. Normal SuccessfulRescale 5m horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target
      25. Normal SuccessfulRescale 13s horizontal-pod-autoscaler New size: 1; reason: All metrics below target
    2. 输入以下命令,确认单个 pod 正在运行:

      1. # kubectl get pods

      你应该会看到类似以下的输出:

      1. NAME READY STATUS RESTARTS AGE
      2. hello-world-54764dfbf8-q6l82 1/1 Running 0 6h