SLO 配置

简介

Koordinator 使用一个 ConfigMap 管理 SLO 配置。该 ConfigMap 被 slo-controller 所使用，它的名字和命名空间可以在 koord-manager 的启 动参数中指定（默认为 koordinator-system/slo-controller-config）。它分别包含了以下键值：

  • colocation-config：混部配置。例如，是否开启混部 Batch 资源，混部水位线。
  • resource-threshold-config：基于阈值的压制/驱逐策略的配置。例如，CPU 压制的阈值，内存驱逐的阈值。
  • resource-qos-config：QoS 特性的配置。例如，BE pods 的 Group Identity，LS pods 的内存 QoS，BE pods 的末级缓存划分。
  • cpu-burst-config：CPU Burst 特性的配置。例如，pod 的最大 burst 比例。
  • system-config：系统设定的配置。例如，全局内存最低水位线系数 min_free_kbytes。

配置层级

每个配置定义为集群级别和节点级别的形式。

例如，

  1. type ColocationCfg struct {
  2. ColocationStrategy `json:",inline"`
  3. NodeConfigs        []NodeColocationCfg `json:"nodeConfigs,omitempty"`
  4. }
  6. type ResourceQOSCfg struct {
  7. ClusterStrategy *slov1alpha1.ResourceQOSStrategy `json:"clusterStrategy,omitempty"`
  8. NodeStrategies  []NodeResourceQOSStrategy        `json:"nodeStrategies,omitempty"`
  9. }

集群级别配置用于设置全局配置，而节点级别则供用户调整部分节点的配置，特别是灰度部署的情况。

请注意，大部分可配置的字段都在组件内部（koordlet、koord-manager）有默认值，所以通常仅需要编辑变更的参数。

NodeSLO

SLO 配置的 data 字段会被 koord-manager 解析。Koord-manager 会检查配置数据是否合法，然后用解析后的配置更新到每个节点的 NodeSLO 对象中。 如果解析失败，koord-manager 会在 ConfigMap 对象上记录 Events，以警示 unmarshal 错误。对于 agent 组件 koordlet，它会 watch NodeSLO 的 Spec，并对节点的 QoS 特性进行调谐。

  1. apiVersion: slo.koordinator.sh/v1alpha1
  2. kind: NodeSLO
  3. metadata:
  4.   name: test-node
  5. spec:
  6.   cpuBurstStrategy: {}
  7.   extensions: {}
  8.   resourceQOSStrategy: {}
  9.   systemStrategy: {}
  10.   # parsed from the `resource-threshold-config` data
  11.   resourceUsedThresholdWithBE:
  12.     cpuSuppressPolicy: cpuset
  13.     cpuSuppressThresholdPercent: 65
  14.     enable: true
  15.     memoryEvictThresholdPercent: 70

配置

参考版本：Koordinator v1.2

SLO 配置的模板如下：

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: slo-controller-config
  5.   namespace: koordinator-system
  6. data:
  7.   # colocation-config is the configuration for colocation.
  8.   # Related features: Dynamic resource over-commitment, Load-aware scheduling, Load-aware descheduling.
  9.   # - enable: whether to enable the colocation. If false, the reclaimed resources of the node allocatable (e.g. `kubernetes.io/batch-cpu`) will be removed.
  10.   # - metricAggregateDurationSeconds: the aggregated duration of node metrics reporting.
  11.   # - metricReportIntervalSeconds: the reporting interval of the node metrics.
  12.   # - metricAggregatePolicy: policies of reporting node metrics in different durations.
  13.   # - cpuReclaimThresholdPercent: the reclaim threshold for calculating the reclaimed cpu resource. Basically, the reclaimed resource cannot reclaim the unused resources which are exceeding the threshold.
  14.   # - memoryReclaimThresholdPercent: the reclaim threshold for calculating the reclaimed memory resource. Basically, the reclaimed resource cannot reclaim the unused resources which are exceeding the threshold.
  15.   # - memoryCalculatePolicy: the policy for calculating the reclaimable memory resource. If set to `request`, only unallocated memory resource of high-priority pods are reclaimable, and no allocated memory can be reclaimed.
  16.   # - degradeTimeMinutes: the threshold duration to degrade the colocation for which the node metrics has not been updated.
  17.   # - updateTimeThresholdSeconds: the threshold duration to force updating the reclaimed resources with the latest calculated result.
  18.   # - resourceDiffThreshold: the threshold to update the reclaimed resources than which the calculated reclaimed resources is different from the current.
  19.   # - nodeConfigs: the node-level configurations which matches the nodes via the node selector and overrides the cluster configuration.
  20.   colocation-config: |
  21.     {
  22.       "enable": false,
  23.       "metricAggregateDurationSeconds": 300,
  24.       "metricReportIntervalSeconds": 60,
  25.       "metricAggregatePolicy": {
  26.         "durations": [
  27.           "5m",
  28.           "10m",
  29.           "15m"
  30.         ]
  31.       },
  32.       "cpuReclaimThresholdPercent": 60,
  33.       "memoryReclaimThresholdPercent": 65,
  34.       "memoryCalculatePolicy": "usage",
  35.       "degradeTimeMinutes": 15,
  36.       "updateTimeThresholdSeconds": 300,
  37.       "resourceDiffThreshold": 0.1,
  38.       "nodeConfigs": [
  39.         {
  40.           "name": "anolis",
  41.           "nodeSelector": {
  42.             "matchLabels": {
  43.               "kubernetes.io/kernel": "anolis"
  44.             }
  45.           },
  46.           "updateTimeThresholdSeconds": 360,
  47.           "resourceDiffThreshold": 0.2
  48.         }
  49.       ]
  50.     }
  51.   # The configuration for threshold-based strategies.
  52.   # Related features: BECPUSuppress, BEMemoryEvict, BECPUEvict.
  53.   # - clusterStrategy: the cluster-level configuration.
  54.   # - nodeStrategies: the node-level configurations which matches the nodes via the node selector and overrides the cluster configuration.
  55.   # - enable: whether to enable the threshold-based strategies or not. If false, all threshold-based strategies are disabled. If set to true, CPU Suppress and Memory Evict are enabled by default.
  56.   # - cpuSuppressThresholdPercent: the node cpu utilization threshold to suppress BE pods' usage.
  57.   # - cpuSuppressPolicy: the policy of cpu suppression. If set to `cpuset`, the BE pods' `cpuset.cpus` will be reconciled when suppression. If set to `cfsQuota`, the BE pods' `cpu.cfs_quota_us` will be reconciled.
  58.   # - memoryEvictThresholdPercent: the node memory utilization threshold to evict BE pods.
  59.   # - memoryEvictLowerPercent: the node memory utilization threshold to stop the memory eviction. By default, `lowerPercent = thresholdPercent - 2`.
  60.   # - cpuEvictBESatisfactionLowerPercent: the cpu satisfaction threshold to start the cpu eviction (also require to meet the BE util threshold).
  61.   # - cpuEvictBEUsageThresholdPercent: the BE utilization (BEUsage / BERealLimit) threshold to start the cpu eviction (also require to meet the cpu satisfaction threshold).
  62.   # - cpuEvictBESatisfactionUpperPercent: the cpu satisfaction threshold to stop the cpu eviction.
  63.   # - cpuEvictTimeWindowSeconds: the time window of the cpu metrics for the cpu eviction.
  64.   resource-threshold-config: |
  65.     {
  66.       "clusterStrategy": {
  67.         "enable": false,
  68.         "cpuSuppressThresholdPercent": 65,
  69.         "cpuSuppressPolicy": "cpuset",
  70.         "memoryEvictThresholdPercent": 70,
  71.         "memoryEvictLowerPercent": 65,
  72.         "cpuEvictBESatisfactionUpperPercent": 90,
  73.         "cpuEvictBESatisfactionLowerPercent": 60,
  74.         "cpuEvictBEUsageThresholdPercent": 90
  75.       },
  76.       "nodeStrategies": [
  77.         {
  78.           "name": "anolis",
  79.           "nodeSelector": {
  80.             "matchLabels": {
  81.               "kubernetes.io/kernel": "anolis"
  82.             }
  83.           },
  84.           "cpuEvictBEUsageThresholdPercent": 80
  85.         }
  86.       ]
  87.     }
  88.   # The configuration for QoS-based features.
  89.   # Related features: CPUQoS (GroupIdentity), MemoryQoS (CgroupReconcile), ResctrlQoS.
  90.   # - clusterStrategy: the cluster-level configuration.
  91.   # - nodeStrategies: the node-level configurations which matches the nodes via the node selector and overrides the cluster configuration.
  92.   # - lsrClass/lsClass/beClass: the configuration for pods of QoS LSR/LS/BE respectively. 
  93.   # - cpuQOS: the configuration of CPU QoS.
  94.   #   - enable: whether to enable CPU QoS. If set to `false`, the related cgroup configs will be reset to the system default.
  95.   #   - groupIdentity: the priority level of the Group Identity ([-1, 2]). `2` means the highest priority, while `-1` means the lowest priority. Anolis OS required.
  96.   # - memoryQOS: the configuration of Memory QoS.
  97.   #   - enable: whether to enable Memory QoS. If set to `false`, the related cgroup configs will be reset to the system default.
  98.   #   - minLimitPercent: the scale percentage for setting the `memory.min` based on the container's request. It enables the memory protection from the Linux memory reclaim.
  99.   #   - lowLimitPercent: the scale percentage for setting the `memory.low` based on the container's request. It enables the memory soft protection from the Linux memory reclaim.
  100.   #   - throttlingPercent: the scale percentage for setting the `memory.high` based on the container's limit. It enables the memory throttling in cgroup level.
  101.   #   - wmarkRatio: the ratio of container-level asynchronous memory reclaim based on the container's limit. Anolis OS required.
  102.   #   - wmarkScalePermill: the per-mill of container memory to reclaim in once asynchronous memory reclaim. Anolis OS required.
  103.   #   - wmarkMinAdj: the adjustment percentage of global memory min watermark. It affects the reclaim priority when the node memory free is quite a few. Anolis OS required.
  104.   # - resctrlQOS: the configuration of Resctrl (Intel RDT) QoS.
  105.   #   - enable: whether to enable Resctrl QoS.
  106.   #   - catRangeStartPercent: the starting percentage of the L3 Cache way partitioning. L3 CAT required.
  107.   #   - catRangeEndPercent: the ending percentage of the L3 Cache way partitioning. L3 CAT required.
  108.   #   - mbaPercent: the allocation percentage of the memory bandwidth. MBA required.
  109.   resource-qos-config: |
  110.     {
  111.       "clusterStrategy": {
  112.         "lsrClass": {
  113.           "cpuQOS": {
  114.             "enable": false,
  115.             "groupIdentity": 2
  116.           },
  117.           "memoryQOS": {
  118.             "enable": false,
  119.             "minLimitPercent": 0,
  120.             "lowLimitPercent": 0,
  121.             "throttlingPercent": 0,
  122.             "wmarkRatio": 95,
  123.             "wmarkScalePermill": 20,
  124.             "wmarkMinAdj": -25,
  125.             "priorityEnable": 0,
  126.             "priority": 0,
  127.             "oomKillGroup": 0
  128.           },
  129.           "resctrlQOS": {
  130.             "enable": false,
  131.             "catRangeStartPercent": 0,
  132.             "catRangeEndPercent": 100,
  133.             "mbaPercent": 100
  134.           }
  135.         },
  136.         "lsClass": {
  137.           "cpuQOS": {
  138.             "enable": false,
  139.             "groupIdentity": 2
  140.           },
  141.           "memoryQOS": {
  142.             "enable": false,
  143.             "minLimitPercent": 0,
  144.             "lowLimitPercent": 0,
  145.             "throttlingPercent": 0,
  146.             "wmarkRatio": 95,
  147.             "wmarkScalePermill": 20,
  148.             "wmarkMinAdj": -25,
  149.             "priorityEnable": 0,
  150.             "priority": 0,
  151.             "oomKillGroup": 0
  152.           },
  153.           "resctrlQOS": {
  154.             "enable": false,
  155.             "catRangeStartPercent": 0,
  156.             "catRangeEndPercent": 100,
  157.             "mbaPercent": 100
  158.           }
  159.         },
  160.         "beClass": {
  161.           "cpuQOS": {
  162.             "enable": false,
  163.             "groupIdentity": -1
  164.           },
  165.           "memoryQOS": {
  166.             "enable": false,
  167.             "minLimitPercent": 0,
  168.             "lowLimitPercent": 0,
  169.             "throttlingPercent": 0,
  170.             "wmarkRatio": 95,
  171.             "wmarkScalePermill": 20,
  172.             "wmarkMinAdj": 50,
  173.             "priorityEnable": 0,
  174.             "priority": 0,
  175.             "oomKillGroup": 0
  176.           },
  177.           "resctrlQOS": {
  178.             "enable": false,
  179.             "catRangeStartPercent": 0,
  180.             "catRangeEndPercent": 30,
  181.             "mbaPercent": 100
  182.           }
  183.         }
  184.       },
  185.       "nodeStrategies": [
  186.         {
  187.           "name": "anolis",
  188.           "nodeSelector": {
  189.             "matchLabels": {
  190.               "kubernetes.io/kernel": "anolis"
  191.             }
  192.           },
  193.           "beClass": {
  194.             "memoryQOS": {
  195.               "wmarkRatio": 90
  196.             }
  197.           }
  198.         }
  199.       ]
  200.     }
  201.   # The configuration for the CPU Burst.
  202.   # Related features: CPUBurst.
  203.   # - clusterStrategy: the cluster-level configuration.
  204.   # - nodeStrategies: the node-level configurations which matches the nodes via the node selector and overrides the cluster configuration.
  205.   # - policy: the policy of CPU Burst. If set to `none`, the CPU Burst is disabled. If set to `auto`, the CPU Burst is fully enabled. If set to `cpuBurstOnly`, only the Linux CFS Burst feature is enabled.
  206.   # - cpuBurstPercent: the percentage of Linux CFS Burst. It affects the value of `cpu.cfs_burst_us` of pod/container cgroups. It specifies the percentage to which the CPU limit can be increased by CPU Burst.
  207.   # - cfsQuotaBurstPercent: the percentage of cfs quota burst. It affects the scaled ratio of `cpu.cfs_quota_us` of pod/container cgroups. It specifies the maximum percentage to which the value of cfs_quota in the cgroup parameters can be increased.
  208.   # - cfsQuotaBurstPeriodSeconds: the maximum period of once cfs quota burst. It indicates that the time period in which the container can run with an increased CFS quota is unlimited.
  209.   # - sharePoolThresholdPercent: the threshold of share pool utilization. If the share pool utilization is too high, CPU Burst will be stopped and reset to avoid machine overload.
  210.   cpu-burst-config: |
  211.     {
  212.       "clusterStrategy": {
  213.         "policy": "none",
  214.         "cpuBurstPercent": 1000,
  215.         "cfsQuotaBurstPercent": 300,
  216.         "cfsQuotaBurstPeriodSeconds": -1,
  217.         "sharePoolThresholdPercent": 50
  218.       },
  219.       "nodeStrategies": [
  220.         {
  221.           "name": "anolis",
  222.           "nodeSelector": {
  223.             "matchLabels": {
  224.               "kubernetes.io/kernel": "anolis"
  225.             }
  226.           },
  227.           "policy": "cfsQuotaBurstOnly",
  228.           "cfsQuotaBurstPercent": 400
  229.         }
  230.       ]
  231.     }
  232.   # The configuration for system-level settings.
  233.   # Related features: SystemConfig.
  234.   # - clusterStrategy: the cluster-level configuration.
  235.   # - nodeStrategies: the node-level configurations which matches the nodes via the node selector and overrides the cluster configuration.
  236.   # - minFreeKbytesFactor: the factor for calculating the global minimum memory free watermark `/proc/sys/vm/min_free_kbytes`. `min_free_kbytes = minFreeKbytesFactor * nodeTotalMemory / 10000`.
  237.   # - watermarkScaleFactor: the reclaim factor `/proc/sys/vm/watermark_scale_factor` in once global memory reclaim.
  238.   # - memcgReapBackGround: whether to enable the reaper for orphan memory cgroups.
  239.   system-config: |-
  240.     {
  241.       "clusterStrategy": {
  242.         "minFreeKbytesFactor": 100,
  243.         "watermarkScaleFactor": 150,
  244.         "memcgReapBackGround": 0
  245.       }
  246.       "nodeStrategies": [
  247.         {
  248.           "name": "anolis",
  249.           "nodeSelector": {
  250.             "matchLabels": {
  251.               "kubernetes.io/kernel": "anolis"
  252.             }
  253.           },
  254.           "minFreeKbytesFactor": 100,
  255.           "watermarkScaleFactor": 150
  256.         }
  257.       ]
  258.     }

对于更多信息，请查看相关特性的用户手册和设计文档。

快速开始

  1. 通过 ConfigMap koordinator-system/slo-controller-config 检查当前的 SLO 配置。
  1. $ kubectl get configmap -n koordinator-system slo-controller-config -o yaml
  2. apiVersion: v1
  3. kind: ConfigMap
  4. metadata:
  5.   annotations:
  6.     meta.helm.sh/release-name: koordinator
  7.     meta.helm.sh/release-namespace: default
  8.   labels:
  9.     app.kubernetes.io/managed-by: Helm
  10.   name: slo-controller-config
  11.   namespace: koordinator-system
  12. data:
  13.   colocation-config: |
  14.     {
  15.       "enable": false,
  16.       "metricAggregateDurationSeconds": 300,
  17.       "metricReportIntervalSeconds": 60,
  18.       "cpuReclaimThresholdPercent": 60,
  19.       "memoryReclaimThresholdPercent": 65,
  20.       "memoryCalculatePolicy": "usage",
  21.       "degradeTimeMinutes": 15,
  22.       "updateTimeThresholdSeconds": 300,
  23.       "resourceDiffThreshold": 0.1
  24.     }
  25.   resource-threshold-config: |
  26.     {
  27.       "clusterStrategy": {
  28.         "enable": false
  29.       }
  30.     }
  1. 编辑 ConfigMap koordinator-system/slo-controller-config 来修改 SLO 配置。
  1. $ kubectl edit configmap -n koordinator-system slo-controller-config

例如，ConfigMap 编辑如下：

  1. data:
  2.   # ...
  3.   resource-threshold-config: |
  4.     {
  5.       "clusterStrategy": {
  6.         "enable": true,
  7.         "cpuSuppressThresholdPercent": 60,
  8.         "cpuSuppressPolicy": "cpuset",
  9.         "memoryEvictThresholdPercent": 60
  10.       }
  11.     }
  1. 确认 NodeSLO 是否成功下发。

注意：默认值会在 NodeSLO 中省略。

  1. $ kubectl get nodeslo.slo.koordinator.sh test-node -o yaml
  2. apiVersion: slo.koordinator.sh/v1alpha1
  3. kind: NodeSLO
  4. metadata:
  5.   name: test-node
  6. spec:
  7.   # ...
  8.   extensions: {}
  9.   resourceUsedThresholdWithBE:
  10.     cpuSuppressPolicy: cpuset
  11.     cpuSuppressThresholdPercent: 60
  12.     enable: true
  13.     memoryEvictThresholdPercent: 60