使用 kubectl 测试 HPA

本文介绍如何在使用负载测试工具扩缩 HPA 后检查 HPA 的状态。有关使用 Rancher UI（最低版本 2.3.x）检查状态的信息，请参阅使用 Rancher UI 管理 HPA。

要让 HPA 正常工作，服务部署应该具有容器的资源请求定义。按照以下 hello-world 示例测试 HPA 是否正常工作。

1. 将 kubectl 连接到你的 Kubernetes 集群。

2. 复制下方的 hello-world 部署清单。

   Hello World 清单

   apiVersion: apps/v1beta2
   kind: Deployment
   metadata:
     labels:
       app: hello-world
     name: hello-world
     namespace: default
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: hello-world
     strategy:
       rollingUpdate:
         maxSurge: 1
         maxUnavailable: 0
       type: RollingUpdate
     template:
       metadata:
         labels:
           app: hello-world
       spec:
         containers:
         - image: rancher/hello-world
           imagePullPolicy: Always
           name: hello-world
           resources:
             requests:
               cpu: 500m
               memory: 64Mi
           ports:
           - containerPort: 80
             protocol: TCP
         restartPolicy: Always
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: hello-world
     namespace: default
   spec:
     ports:
     - port: 80
       protocol: TCP
       targetPort: 80
     selector:
       app: hello-world

3. 将其部署到你的集群。

   # kubectl create -f <HELLO_WORLD_MANIFEST>

4. 根据你使用的指标类型复制以下其中一个 HPA：

   Hello World HPA：资源指标

   apiVersion: autoscaling/v2beta1
   kind: HorizontalPodAutoscaler
   metadata:
     name: hello-world
     namespace: default
   spec:
     scaleTargetRef:
       apiVersion: extensions/v1beta1
       kind: Deployment
       name: hello-world
     minReplicas: 1
     maxReplicas: 10
     metrics:
     - type: Resource
       resource:
         name: cpu
         targetAverageUtilization: 50
     - type: Resource
       resource:
         name: memory
         targetAverageValue: 1000Mi

   Hello World HPA：自定义指标

   apiVersion: autoscaling/v2beta1
   kind: HorizontalPodAutoscaler
   metadata:
     name: hello-world
     namespace: default
   spec:
     scaleTargetRef:
       apiVersion: extensions/v1beta1
       kind: Deployment
       name: hello-world
     minReplicas: 1
     maxReplicas: 10
     metrics:
     - type: Resource
       resource:
         name: cpu
         targetAverageUtilization: 50
     - type: Resource
       resource:
         name: memory
         targetAverageValue: 100Mi
     - type: Pods
       pods:
         metricName: cpu_system
         targetAverageValue: 20m

5. 查看 HPA 信息和描述。确认显示的指标数据。

   资源指标

   1. 输入以下命令：

      # kubectl get hpa
      NAME          REFERENCE                TARGETS                     MINPODS   MAXPODS   REPLICAS   AGE
      hello-world   Deployment/hello-world   1253376 / 100Mi, 0% / 50%   1         10        1          6m
      # kubectl describe hpa
      Name:                                                  hello-world
      Namespace:                                             default
      Labels:                                                <none>
      Annotations:                                           <none>
      CreationTimestamp:                                     Mon, 23 Jul 2018 20:21:16 +0200
      Reference:                                             Deployment/hello-world
      Metrics:                                               ( current / target )
        resource memory on pods:                             1253376 / 100Mi
        resource cpu on pods  (as a percentage of request):  0% (0) / 50%
      Min replicas:                                          1
      Max replicas:                                          10
      Conditions:
        Type            Status  Reason              Message
        ----            ------  ------              -------
        AbleToScale     True    ReadyForNewScale    the last scale time was sufficiently old as to warrant a new scale
        ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource
        ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
      Events:           <none>

      自定义指标

   2. 输入以下命令：

      # kubectl describe hpa

      你应该会看到类似以下的输出：

      Name:                                                  hello-world
      Namespace:                                             default
      Labels:                                                <none>
      Annotations:                                           <none>
      CreationTimestamp:                                     Tue, 24 Jul 2018 18:36:28 +0200
      Reference:                                             Deployment/hello-world
      Metrics:                                               ( current / target )
        resource memory on pods:                             3514368 / 100Mi
        "cpu_system" on pods:                                0 / 20m
        resource cpu on pods  (as a percentage of request):  0% (0) / 50%
      Min replicas:                                          1
      Max replicas:                                          10
      Conditions:
        Type            Status  Reason              Message
        ----            ------  ------              -------
        AbleToScale     True    ReadyForNewScale    the last scale time was sufficiently old as to warrant a new scale
        ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource
        ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
      Events:           <none>

6. 为服务生成负载，从而测试你的 pod 是否按预期进行了自动扩缩。你可以使用任何负载测试工具（Hey、Gatling 等），我们使用的是 Hey。

7. 测试 pod 自动扩缩是否按预期工作。

   使用资源指标测试自动扩缩：

   扩展到两个 Pod：CPU 用量达到目标

   使用你的负载测试工具根据 CPU 使用情况扩展到两个 Pod。

   1. 查看你的 HPA。

      # kubectl describe hpa

      你应该会看到类似以下的输出：

      Name:                                                  hello-world
      Namespace:                                             default
      Labels:                                                <none>
      Annotations:                                           <none>
      CreationTimestamp:                                     Mon, 23 Jul 2018 22:22:04 +0200
      Reference:                                             Deployment/hello-world
      Metrics:                                               ( current / target )
        resource memory on pods:                             10928128 / 100Mi
        resource cpu on pods  (as a percentage of request):  56% (280m) / 50%
      Min replicas:                                          1
      Max replicas:                                          10
      Conditions:
        Type            Status  Reason              Message
        ----            ------  ------              -------
        AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 2
        ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
        ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
      Events:
        Type    Reason             Age   From                       Message
        ----    ------             ----  ----                       -------
        Normal  SuccessfulRescale  13s   horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target

   2. 输入以下命令，确认你已扩展到两个 pod：

      # kubectl get pods

      你应该会看到类似以下的输出：

         NAME                                                     READY     STATUS    RESTARTS   AGE
         hello-world-54764dfbf8-k8ph2                             1/1       Running   0          1m
         hello-world-54764dfbf8-q6l4v                             1/1       Running   0          3h

      扩展到三个 Pod：CPU 用量达到目标

   使用你的负载测试工具根据 CPU 用量扩展到三个 Pod，并将 horizo​​ntal-pod-autoscaler-upscale-delay 设置为 3 分钟。

   1. 输入以下命令：

      # kubectl describe hpa

      你应该会看到类似以下的输出：

         Name:                                                  hello-world
         Namespace:                                             default
         Labels:                                                <none>
         Annotations:                                           <none>
         CreationTimestamp:                                     Mon, 23 Jul 2018 22:22:04 +0200
         Reference:                                             Deployment/hello-world
         Metrics:                                               ( current / target )
           resource memory on pods:                             9424896 / 100Mi
           resource cpu on pods  (as a percentage of request):  66% (333m) / 50%
         Min replicas:                                          1
         Max replicas:                                          10
         Conditions:
           Type            Status  Reason              Message
           ----            ------  ------              -------
           AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 3
           ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
           ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
         Events:
           Type    Reason             Age   From                       Message
           ----    ------             ----  ----                       -------
           Normal  SuccessfulRescale  4m    horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
           Normal  SuccessfulRescale  16s   horizontal-pod-autoscaler  New size: 3; reason: cpu resource utilization (percentage of request) above target

   2. 输入以下命令，确认三个 pod 正在运行：

      # kubectl get pods

      你应该会看到类似以下的输出：

      NAME                                                     READY     STATUS    RESTARTS   AGE
      hello-world-54764dfbf8-f46kh                             0/1       Running   0          1m
      hello-world-54764dfbf8-k8ph2                             1/1       Running   0          5m
      hello-world-54764dfbf8-q6l4v                             1/1       Running   0          3h

      缩减到 1 个 Pod：所有指标均低于目标

   当 horizontal-pod-autoscaler-downscale-delay 的所有指标均低于目标（默认为 5 分钟）时，使用你的负载测试工具缩减到 1 个 pod。

   1. 输入以下命令：

      # kubectl describe hpa

      你应该会看到类似以下的输出：

          Name:                                                  hello-world
          Namespace:                                             default
          Labels:                                                <none>
          Annotations:                                           <none>
          CreationTimestamp:                                     Mon, 23 Jul 2018 22:22:04 +0200
          Reference:                                             Deployment/hello-world
          Metrics:                                               ( current / target )
            resource memory on pods:                             10070016 / 100Mi
            resource cpu on pods  (as a percentage of request):  0% (0) / 50%
          Min replicas:                                          1
          Max replicas:                                          10
          Conditions:
            Type            Status  Reason              Message
            ----            ------  ------              -------
            AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 1
            ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource
            ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
          Events:
            Type    Reason             Age   From                       Message
            ----    ------             ----  ----                       -------
            Normal  SuccessfulRescale  10m   horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
            Normal  SuccessfulRescale  6m    horizontal-pod-autoscaler  New size: 3; reason: cpu resource utilization (percentage of request) above target
            Normal  SuccessfulRescale  1s    horizontal-pod-autoscaler  New size: 1; reason: All metrics below target

   使用自定义指标测试自动缩放：

   扩展到两个 Pod：CPU 用量达到目标

   使用负载测试工具根据 CPU 用量扩展到两个 Pod。

   1. 输入以下命令：

        # kubectl describe hpa

      你应该会看到类似以下的输出：

        Name:                                                  hello-world
        Namespace:                                             default
        Labels:                                                <none>
        Annotations:                                           <none>
        CreationTimestamp:                                     Tue, 24 Jul 2018 18:01:11 +0200
        Reference:                                             Deployment/hello-world
        Metrics:                                               ( current / target )
          resource memory on pods:                             8159232 / 100Mi
          "cpu_system" on pods:                                7m / 20m
          resource cpu on pods  (as a percentage of request):  64% (321m) / 50%
        Min replicas:                                          1
        Max replicas:                                          10
        Conditions:
          Type            Status  Reason              Message
          ----            ------  ------              -------
          AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 2
          ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
          ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
        Events:
          Type    Reason             Age   From                       Message
          ----    ------             ----  ----                       -------
          Normal  SuccessfulRescale  16s   horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target

   2. 输入以下命令，确认两个 pod 正在运行：

        # kubectl get pods

      你应该会看到类似以下的输出：

            NAME                           READY     STATUS    RESTARTS   AGE
            hello-world-54764dfbf8-5pfdr   1/1       Running   0          3s
            hello-world-54764dfbf8-q6l82   1/1       Running   0          6h

      扩展到三个 Pod：CPU 用量达到目标

   当 cpu_system 用量达到目标时，使用你的负载测试工具扩展到三个 Pod。

   1. 输入以下命令：

      # kubectl describe hpa

      你应该会看到类似以下的输出：

         Name:                                                  hello-world
         Namespace:                                             default
         Labels:                                                <none>
         Annotations:                                           <none>
         CreationTimestamp:                                     Tue, 24 Jul 2018 18:01:11 +0200
         Reference:                                             Deployment/hello-world
         Metrics:                                               ( current / target )
           resource memory on pods:                             8374272 / 100Mi
           "cpu_system" on pods:                                27m / 20m
           resource cpu on pods  (as a percentage of request):  71% (357m) / 50%
         Min replicas:                                          1
         Max replicas:                                          10
         Conditions:
           Type            Status  Reason              Message
           ----            ------  ------              -------
           AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 3
           ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
           ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
         Events:
           Type    Reason             Age   From                       Message
           ----    ------             ----  ----                       -------
           Normal  SuccessfulRescale  3m    horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
           Normal  SuccessfulRescale  3s    horizontal-pod-autoscaler  New size: 3; reason: pods metric cpu_system above target

   2. 输入以下命令，确认三个 pod 正在运行：

      # kubectl get pods

      你应该会看到类似以下的输出：

         # kubectl get pods
         NAME                           READY     STATUS    RESTARTS   AGE
         hello-world-54764dfbf8-5pfdr   1/1       Running   0          3m
         hello-world-54764dfbf8-m2hrl   1/1       Running   0          1s
         hello-world-54764dfbf8-q6l82   1/1       Running   0          6h

      扩展到四个 Pod：CPU 用量达到目标

   使用负载测试工具根据 CPU 用量扩展到四个 Pod。horizontal-pod-autoscaler-upscale-delay 默认设置为 3 分钟。

   1. 输入以下命令：

      # kubectl describe hpa

      你应该会看到类似以下的输出：

          Name:                                                  hello-world
          Namespace:                                             default
          Labels:                                                <none>
          Annotations:                                           <none>
          CreationTimestamp:                                     Tue, 24 Jul 2018 18:01:11 +0200
          Reference:                                             Deployment/hello-world
          Metrics:                                               ( current / target )
            resource memory on pods:                             8374272 / 100Mi
            "cpu_system" on pods:                                27m / 20m
            resource cpu on pods  (as a percentage of request):  71% (357m) / 50%
          Min replicas:                                          1
          Max replicas:                                          10
          Conditions:
            Type            Status  Reason              Message
            ----            ------  ------              -------
            AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 3
            ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
            ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
          Events:
            Type    Reason             Age   From                       Message
            ----    ------             ----  ----                       -------
            Normal  SuccessfulRescale  5m    horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
            Normal  SuccessfulRescale  3m    horizontal-pod-autoscaler  New size: 3; reason: pods metric cpu_system above target
            Normal  SuccessfulRescale  4s    horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target

   2. 输入以下命令，确认四个 pod 正在运行：

      # kubectl get pods

      你应该会看到类似以下的输出：

        NAME                           READY     STATUS    RESTARTS   AGE
        hello-world-54764dfbf8-2p9xb   1/1       Running   0          5m
        hello-world-54764dfbf8-5pfdr   1/1       Running   0          2m
        hello-world-54764dfbf8-m2hrl   1/1       Running   0          1s
        hello-world-54764dfbf8-q6l82   1/1       Running   0          6h

      缩减到 1 个 Pod：所有指标均低于目标

   当 horizontal-pod-autoscaler-downscale-delay 的所有指标均低于目标时，使用你的负载测试工具缩减到 1 个 pod。

   1. 输入以下命令：

      # kubectl describe hpa

      你应该会看到类似以下的输出：

          Name:                                                  hello-world
          Namespace:                                             default
          Labels:                                                <none>
          Annotations:                                           <none>
          CreationTimestamp:                                     Tue, 24 Jul 2018 18:01:11 +0200
          Reference:                                             Deployment/hello-world
          Metrics:                                               ( current / target )
            resource memory on pods:                             8101888 / 100Mi
            "cpu_system" on pods:                                8m / 20m
            resource cpu on pods  (as a percentage of request):  0% (0) / 50%
          Min replicas:                                          1
          Max replicas:                                          10
          Conditions:
            Type            Status  Reason              Message
            ----            ------  ------              -------
            AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 1
            ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource
            ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
          Events:
            Type    Reason             Age   From                       Message
            ----    ------             ----  ----                       -------
            Normal  SuccessfulRescale  10m    horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
            Normal  SuccessfulRescale  8m    horizontal-pod-autoscaler  New size: 3; reason: pods metric cpu_system above target
            Normal  SuccessfulRescale  5m    horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
            Normal   SuccessfulRescale             13s               horizontal-pod-autoscaler  New size: 1; reason: All metrics below target

   2. 输入以下命令，确认单个 pod 正在运行：

          # kubectl get pods

      你应该会看到类似以下的输出：

          NAME                           READY     STATUS    RESTARTS   AGE
          hello-world-54764dfbf8-q6l82   1/1       Running   0          6h