通过 Kubectl 测试 HPA

本文档介绍了使用负载测试工具触发 HPA 按比例扩容或缩容后如何检查其状态。有关如何从 Rancher UI(至少 2.3.x 版)检查状态的信息,请参阅 使用 Rancher UI 管理 HPA

为了使 HPA 正常工作,服务部署应具有容器的资源请求(request)定义。可以按照此 hello-world 示例测试 HPA 是否正常工作。

配置 kubectl 以连接到您的 Kubernetes 集群。

复制下面的 hello-world 部署清单。

  1. apiVersion: apps/v1beta2
  2. kind: Deployment
  3. metadata:
  4. labels:
  5. app: hello-world
  6. name: hello-world
  7. namespace: default
  8. spec:
  9. replicas: 1
  10. selector:
  11. matchLabels:
  12. app: hello-world
  13. strategy:
  14. rollingUpdate:
  15. maxSurge: 1
  16. maxUnavailable: 0
  17. type: RollingUpdate
  18. template:
  19. metadata:
  20. labels:
  21. app: hello-world
  22. spec:
  23. containers:
  24. - image: rancher/hello-world
  25. imagePullPolicy: Always
  26. name: hello-world
  27. resources:
  28. requests:
  29. cpu: 500m
  30. memory: 64Mi
  31. ports:
  32. - containerPort: 80
  33. protocol: TCP
  34. restartPolicy: Always
  35. ---
  36. apiVersion: v1
  37. kind: Service
  38. metadata:
  39. name: hello-world
  40. namespace: default
  41. spec:
  42. ports:
  43. - port: 80
  44. protocol: TCP
  45. targetPort: 80
  46. selector:
  47. app: hello-world

将其部署到您的集群。

  1. # kubectl create -f <HELLO_WORLD_MANIFEST>

根据您使用的指标类型复制下面适合您的 HPA:

  • Hello World HPA: 资源指标

    1. apiVersion: autoscaling/v2beta1
    2. kind: HorizontalPodAutoscaler
    3. metadata:
    4. name: hello-world
    5. namespace: default
    6. spec:
    7. scaleTargetRef:
    8. apiVersion: extensions/v1beta1
    9. kind: Deployment
    10. name: hello-world
    11. minReplicas: 1
    12. maxReplicas: 10
    13. metrics:
    14. - type: Resource
    15. resource:
    16. name: cpu
    17. targetAverageUtilization: 50
    18. - type: Resource
    19. resource:
    20. name: memory
    21. targetAverageValue: 1000Mi
  • Hello World HPA: 自定义指标

    1. apiVersion: autoscaling/v2beta1
    2. kind: HorizontalPodAutoscaler
    3. metadata:
    4. name: hello-world
    5. namespace: default
    6. spec:
    7. scaleTargetRef:
    8. apiVersion: extensions/v1beta1
    9. kind: Deployment
    10. name: hello-world
    11. minReplicas: 1
    12. maxReplicas: 10
    13. metrics:
    14. - type: Resource
    15. resource:
    16. name: cpu
    17. targetAverageUtilization: 50
    18. - type: Resource
    19. resource:
    20. name: memory
    21. targetAverageValue: 100Mi
    22. - type: Pods
    23. pods:
    24. metricName: cpu_system
    25. targetAverageValue: 20m

查看 HPA 信息和说明。确认已显示指标数据。

  • 资源指标

    1. 输入以下命令。

      1. # kubectl get hpa
      2. NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
      3. hello-world Deployment/hello-world 1253376 / 100Mi, 0% / 50% 1 10 1 6m
      4. # kubectl describe hpa
      5. Name: hello-world
      6. Namespace: default
      7. Labels: <none>
      8. Annotations: <none>
      9. CreationTimestamp: Mon, 23 Jul 2018 20:21:16 +0200
      10. Reference: Deployment/hello-world
      11. Metrics: ( current / target )
      12. resource memory on pods: 1253376 / 100Mi
      13. resource cpu on pods (as a percentage of request): 0% (0) / 50%
      14. Min replicas: 1
      15. Max replicas: 10
      16. Conditions:
      17. Type Status Reason Message
      18. ---- ------ ------ -------
      19. AbleToScale True ReadyForNewScale the last scale time was sufficiently old as to warrant a new scale
      20. ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from memory resource
      21. ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
      22. Events: <none>
  • 自定义指标

    1. 输入以下命令。

      1. # kubectl describe hpa
    2. 您应该看到以下输出:

      1. Name: hello-world
      2. Namespace: default
      3. Labels: <none>
      4. Annotations: <none>
      5. CreationTimestamp: Tue, 24 Jul 2018 18:36:28 +0200
      6. Reference: Deployment/hello-world
      7. Metrics: ( current / target )
      8. resource memory on pods: 3514368 / 100Mi
      9. "cpu_system" on pods: 0 / 20m
      10. resource cpu on pods (as a percentage of request): 0% (0) / 50%
      11. Min replicas: 1
      12. Max replicas: 10
      13. Conditions:
      14. Type Status Reason Message
      15. ---- ------ ------ -------
      16. AbleToScale True ReadyForNewScale the last scale time was sufficiently old as to warrant a new scale
      17. ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from memory resource
      18. ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
      19. Events: <none>

对 HPA 服务进行压测,以测试您的 Pod 是否可以按预期自动缩放。

您可以使用任何负载测试工具(Hey,Gatling 等),我们使用的是 Hey

测试 Pod 自动扩缩容功能是否按预期工作。

使用资源指标测试自动扩缩容

扩容到 2 个 Pod:CPU 使用量超过目标值

使用负载测试工具对工作负载进行压测,提高 CPU 使用率触发自动扩缩容,使得 pod 数量扩容至 2 个。

  1. 查看您的 HPA。

    1. # kubectl describe hpa

    您应该收到与以下类似的输出。

    1. Name: hello-world
    2. Namespace: default
    3. Labels: <none>
    4. Annotations: <none>
    5. CreationTimestamp: Mon, 23 Jul 2018 22:22:04 +0200
    6. Reference: Deployment/hello-world
    7. Metrics: ( current / target )
    8. resource memory on pods: 10928128 / 100Mi
    9. resource cpu on pods (as a percentage of request): 56% (280m) / 50%
    10. Min replicas: 1
    11. Max replicas: 10
    12. Conditions:
    13. Type Status Reason Message
    14. ---- ------ ------ -------
    15. AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 2
    16. ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
    17. ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
    18. Events:
    19. Type Reason Age From Message
    20. ---- ------ ---- ---- -------
    21. Normal SuccessfulRescale 13s horizontal-pod-autoscaler New size: 2; reason: cpu resource utilization (percentage of request) above target
  2. 输入以下命令以确认您已扩容至 2 个 pod。

    1. # kubectl get pods

    您应该收到类似于以下内容的输出:

    1. NAME READY STATUS RESTARTS AGE
    2. hello-world-54764dfbf8-k8ph2 1/1 Running 0 1m
    3. hello-world-54764dfbf8-q6l4v 1/1 Running 0 3h

扩容到 3 个 Pod:CPU 使用量超过目标值

使用您的负载测试工具,根据 CPU 使用率将 horizontal-pod-autoscaler-upscale-delay 设置为 3 分钟,以将其扩展到 3 个 pod。

  1. 输入以下命令。

    1. # kubectl describe hpa

    您应该收到类似于以下内容的输出

    1. Name: hello-world
    2. Namespace: default
    3. Labels: <none>
    4. Annotations: <none>
    5. CreationTimestamp: Mon, 23 Jul 2018 22:22:04 +0200
    6. Reference: Deployment/hello-world
    7. Metrics: ( current / target )
    8. resource memory on pods: 9424896 / 100Mi
    9. resource cpu on pods (as a percentage of request): 66% (333m) / 50%
    10. Min replicas: 1
    11. Max replicas: 10
    12. Conditions:
    13. Type Status Reason Message
    14. ---- ------ ------ -------
    15. AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 3
    16. ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
    17. ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
    18. Events:
    19. Type Reason Age From Message
    20. ---- ------ ---- ---- -------
    21. Normal SuccessfulRescale 4m horizontal-pod-autoscaler New size: 2; reason: cpu resource utilization (percentage of request) above target
    22. Normal SuccessfulRescale 16s horizontal-pod-autoscaler New size: 3; reason: cpu resource utilization (percentage of request) above target
  2. 输入以下命令以确认您已扩容至 3 个 pod。

    1. # kubectl get pods

    您应该收到与以下类似的输出。

    1. NAME READY STATUS RESTARTS AGE
    2. hello-world-54764dfbf8-f46kh 0/1 Running 0 1m
    3. hello-world-54764dfbf8-k8ph2 1/1 Running 0 5m
    4. hello-world-54764dfbf8-q6l4v 1/1 Running 0 3h

缩容到 1 个 Pod: 全部指标低于目标值

当所有指标均低于“ horizontal-pod-autoscaler-downscale-delay”的目标(默认为 5 分钟)时,请使用压测工具将其缩小到 1 个 pod。

  1. 输入以下命令。

    1. # kubectl describe hpa

    您应该收到与以下类似的输出。

    1. Name: hello-world
    2. Namespace: default
    3. Labels: <none>
    4. Annotations: <none>
    5. CreationTimestamp: Mon, 23 Jul 2018 22:22:04 +0200
    6. Reference: Deployment/hello-world
    7. Metrics: ( current / target )
    8. resource memory on pods: 10070016 / 100Mi
    9. resource cpu on pods (as a percentage of request): 0% (0) / 50%
    10. Min replicas: 1
    11. Max replicas: 10
    12. Conditions:
    13. Type Status Reason Message
    14. ---- ------ ------ -------
    15. AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 1
    16. ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from memory resource
    17. ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
    18. Events:
    19. Type Reason Age From Message
    20. ---- ------ ---- ---- -------
    21. Normal SuccessfulRescale 10m horizontal-pod-autoscaler New size: 2; reason: cpu resource utilization (percentage of request) above target
    22. Normal SuccessfulRescale 6m horizontal-pod-autoscaler New size: 3; reason: cpu resource utilization (percentage of request) above target
    23. Normal SuccessfulRescale 1s horizontal-pod-autoscaler New size: 1; reason: All metrics below target

使用自定义指标测试自动扩缩容

扩容到 2 个 Pod:CPU 使用量超过目标值

使用您的负载测试工具加大应用负载,以触发 HPA 根据 CPU 使用率自动扩容至 2 个 Pod。

  1. 输入以下命令。

    1. # kubectl describe hpa

    您应该收到与以下类似的输出。

    1. Name: hello-world
    2. Namespace: default
    3. Labels: <none>
    4. Annotations: <none>
    5. CreationTimestamp: Tue, 24 Jul 2018 18:01:11 +0200
    6. Reference: Deployment/hello-world
    7. Metrics: ( current / target )
    8. resource memory on pods: 8159232 / 100Mi
    9. "cpu_system" on pods: 7m / 20m
    10. resource cpu on pods (as a percentage of request): 64% (321m) / 50%
    11. Min replicas: 1
    12. Max replicas: 10
    13. Conditions:
    14. Type Status Reason Message
    15. ---- ------ ------ -------
    16. AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 2
    17. ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
    18. ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
    19. Events:
    20. Type Reason Age From Message
    21. ---- ------ ---- ---- -------
    22. Normal SuccessfulRescale 16s horizontal-pod-autoscaler New size: 2; reason: cpu resource utilization (percentage of request) above target
  2. 输入以下命令以确认两个 Pod 正在运行。

    1. # kubectl get pods

    您应该收到与以下类似的输出。

    1. NAME READY STATUS RESTARTS AGE
    2. hello-world-54764dfbf8-5pfdr 1/1 Running 0 3s
    3. hello-world-54764dfbf8-q6l82 1/1 Running 0 6h

扩容到 3 个 Pod:CPU 使用量超过目标值

使用负载测试工具进行压测,当 cpu_system 的限制达到目标值时,pod 数量会继续扩容至 3 个

  1. 输入以下命令。

    1. # kubectl describe hpa

    您应该收到类似于以下内容的输出:

    1. Name: hello-world
    2. Namespace: default
    3. Labels: <none>
    4. Annotations: <none>
    5. CreationTimestamp: Tue, 24 Jul 2018 18:01:11 +0200
    6. Reference: Deployment/hello-world
    7. Metrics: ( current / target )
    8. resource memory on pods: 8374272 / 100Mi
    9. "cpu_system" on pods: 27m / 20m
    10. resource cpu on pods (as a percentage of request): 71% (357m) / 50%
    11. Min replicas: 1
    12. Max replicas: 10
    13. Conditions:
    14. Type Status Reason Message
    15. ---- ------ ------ -------
    16. AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 3
    17. ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
    18. ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
    19. Events:
    20. Type Reason Age From Message
    21. ---- ------ ---- ---- -------
    22. Normal SuccessfulRescale 3m horizontal-pod-autoscaler New size: 2; reason: cpu resource utilization (percentage of request) above target
    23. Normal SuccessfulRescale 3s horizontal-pod-autoscaler New size: 3; reason: pods metric cpu_system above target
  2. 输入以下命令以确认三个 Pod 正在运行。

    1. # kubectl get pods

    您应该收到类似于以下内容的输出:

    1. # kubectl get pods
    2. NAME READY STATUS RESTARTS AGE
    3. hello-world-54764dfbf8-5pfdr 1/1 Running 0 3m
    4. hello-world-54764dfbf8-m2hrl 1/1 Running 0 1s
    5. hello-world-54764dfbf8-q6l82 1/1 Running 0 6h

扩容到 4 个 Pod:CPU 使用量超过目标值

使用您的负载测试工具进行压测,提升 CPU 使用率,触发 HPA 机制扩容至 4 个 Pod。默认情况下, horizontal-pod-autoscaler-upscale-delay 设置为三分钟。

  1. 输入以下命令。

    1. # kubectl describe hpa

    您应该收到与以下类似的输出。

    1. Name: hello-world
    2. Namespace: default
    3. Labels: <none>
    4. Annotations: <none>
    5. CreationTimestamp: Tue, 24 Jul 2018 18:01:11 +0200
    6. Reference: Deployment/hello-world
    7. Metrics: ( current / target )
    8. resource memory on pods: 8374272 / 100Mi
    9. "cpu_system" on pods: 27m / 20m
    10. resource cpu on pods (as a percentage of request): 71% (357m) / 50%
    11. Min replicas: 1
    12. Max replicas: 10
    13. Conditions:
    14. Type Status Reason Message
    15. ---- ------ ------ -------
    16. AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 3
    17. ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
    18. ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
    19. Events:
    20. Type Reason Age From Message
    21. ---- ------ ---- ---- -------
    22. Normal SuccessfulRescale 5m horizontal-pod-autoscaler New size: 2; reason: cpu resource utilization (percentage of request) above target
    23. Normal SuccessfulRescale 3m horizontal-pod-autoscaler New size: 3; reason: pods metric cpu_system above target
    24. Normal SuccessfulRescale 4s horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target
  2. 输入以下命令以确认四个 Pod 正在运行。

    1. # kubectl get pods

    您应该收到与以下类似的输出。

    1. NAME READY STATUS RESTARTS AGE
    2. hello-world-54764dfbf8-2p9xb 1/1 Running 0 5m
    3. hello-world-54764dfbf8-5pfdr 1/1 Running 0 2m
    4. hello-world-54764dfbf8-m2hrl 1/1 Running 0 1s
    5. hello-world-54764dfbf8-q6l82 1/1 Running 0 6h

缩容到 1 个 Pod: 全部指标低于目标值

当所有指标均低于 horizontal-pod-autoscaler-downscale-delay 目标值时,请使用负载测试工具将其缩小到 1 个 Pod。

  1. 输入以下命令。

    1. # kubectl describe hpa

    您应该收到与以下类似的输出。

    1. Name: hello-world
    2. Namespace: default
    3. Labels: <none>
    4. Annotations: <none>
    5. CreationTimestamp: Tue, 24 Jul 2018 18:01:11 +0200
    6. Reference: Deployment/hello-world
    7. Metrics: ( current / target )
    8. resource memory on pods: 8101888 / 100Mi
    9. "cpu_system" on pods: 8m / 20m
    10. resource cpu on pods (as a percentage of request): 0% (0) / 50%
    11. Min replicas: 1
    12. Max replicas: 10
    13. Conditions:
    14. Type Status Reason Message
    15. ---- ------ ------ -------
    16. AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 1
    17. ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from memory resource
    18. ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
    19. Events:
    20. Type Reason Age From Message
    21. ---- ------ ---- ---- -------
    22. Normal SuccessfulRescale 10m horizontal-pod-autoscaler New size: 2; reason: cpu resource utilization (percentage of request) above target
    23. Normal SuccessfulRescale 8m horizontal-pod-autoscaler New size: 3; reason: pods metric cpu_system above target
    24. Normal SuccessfulRescale 5m horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target
    25. Normal SuccessfulRescale 13s horizontal-pod-autoscaler New size: 1; reason: All metrics below target
  2. 输入以下命令以确认单个 Pod 正在运行。

    1. # kubectl get pods

    您应该收到与以下类似的输出。

    1. NAME READY STATUS RESTARTS AGE
    2. hello-world-54764dfbf8-q6l82 1/1 Running 0 6h