在联邦中设置放置策略

已过时

强烈建议不要使用联邦 v1 版本联邦 v1 版本从未达到 GA 状态,且不再处于积极开发阶段。文档仅作为历史参考。

有关更多信息,请参阅预期的替代品 Kubernetes 联邦 v2 版本

此页面显示如何使用外部策略引擎对联邦资源强制执行基于策略的放置决策。

准备开始

您需要一个正在运行的 Kubernetes 集群(它被引用为主机集群)。有关您的平台的安装说明,请参阅入门指南。

Deploying 联邦并配置外部策略引擎

可以使用 kubefed init 部署联邦控制平面。

Deploying 联邦控制平面之后,必须在联邦 API 服务器中配置一个准入控制器,该控制器强制执行从外部策略引擎接收到的放置决策。

  1. kubectl create -f scheduling-policy-admission.yaml

下图是准入控制器的 ConfigMap 示例:

federation/scheduling-policy-admission.yaml 在联邦中设置放置策略 - 图1
  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4. name: admission
  5. namespace: federation-system
  6. data:
  7. config.yml: |
  8. apiVersion: apiserver.k8s.io/v1alpha1
  9. kind: AdmissionConfiguration
  10. plugins:
  11. - name: SchedulingPolicy
  12. path: /etc/kubernetes/admission/scheduling-policy-config.yml
  13. scheduling-policy-config.yml: |
  14. kubeconfig: /etc/kubernetes/admission/opa-kubeconfig
  15. opa-kubeconfig: |
  16. clusters:
  17. - name: opa-api
  18. cluster:
  19. server: http://opa.federation-system.svc.cluster.local:8181/v0/data/kubernetes/placement
  20. users:
  21. - name: scheduling-policy
  22. user:
  23. token: deadbeefsecret
  24. contexts:
  25. - name: default
  26. context:
  27. cluster: opa-api
  28. user: scheduling-policy
  29. current-context: default

ConfigMap 包含三个文件:

  • config.yml 指定 调度策略 准入控制器配置文件的位置。
  • scheduling-policy-config.yml 指定与外部策略引擎联系所需的 kubeconfig 文件的位置。 该文件还可以包含一个 retryBackoff 值,该值以毫秒为单位控制初始重试 backoff 延迟。
  • opa-kubeconfig 是一个标准的 kubeconfig,包含联系外部策略引擎所需的 URL 和凭证。

编辑联邦 API 服务器部署以启用 SchedulingPolicy 准入控制器。

  1. kubectl -n federation-system edit deployment federation-apiserver

更新 Federation API 服务器命令行参数以启用准入控制器, 并将 ConfigMap 挂载到容器中。如果存在现有的 -enable-admissionplugins 参数,则追加 SchedulingPolicy 而不是添加另一行。

  1. --enable-admission-plugins=SchedulingPolicy
  2. --admission-control-config-file=/etc/kubernetes/admission/config.yml

将以下卷添加到联邦 API 服务器 pod:

  1. - name: admission-config
  2. configMap:
  3. name: admission

添加以下卷挂载联邦 API 服务器的 apiserver 容器:

  1. volumeMounts:
  2. - name: admission-config
  3. mountPath: /etc/kubernetes/admission

Deploying 外部策略引擎

Open Policy Agent (OPA) 是一个开源的通用策略引擎, 您可以使用它在联邦控制平面中执行基于策略的放置决策。

在主机群集中创建服务以联系外部策略引擎:

  1. kubectl create -f policy-engine-service.yaml

下面显示的是 OPA 的示例服务。

federation/policy-engine-service.yaml 在联邦中设置放置策略 - 图2
  1. kind: Service
  2. apiVersion: v1
  3. metadata:
  4. name: opa
  5. namespace: federation-system
  6. spec:
  7. selector:
  8. app: opa
  9. ports:
  10. - name: http
  11. protocol: TCP
  12. port: 8181
  13. targetPort: 8181

使用联邦控制平面在主机群集中创建部署:

  1. kubectl create -f policy-engine-deployment.yaml

下面显示的是 OPA 的部署示例。

federation/policy-engine-deployment.yaml 在联邦中设置放置策略 - 图3
  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4. labels:
  5. app: opa
  6. name: opa
  7. namespace: federation-system
  8. spec:
  9. replicas: 1
  10. selector:
  11. matchLabels:
  12. app: opa
  13. template:
  14. metadata:
  15. labels:
  16. app: opa
  17. name: opa
  18. spec:
  19. containers:
  20. - name: opa
  21. image: openpolicyagent/opa:0.4.10
  22. args:
  23. - run
  24. - “—server
  25. - name: kube-mgmt
  26. image: openpolicyagent/kube-mgmt:0.2
  27. args:
  28. - “-kubeconfig=/srv/kubernetes/kubeconfig
  29. - “-cluster=federation/v1beta1/clusters
  30. volumeMounts:
  31. - name: federation-kubeconfig
  32. mountPath: /srv/kubernetes
  33. readOnly: true
  34. volumes:
  35. - name: federation-kubeconfig
  36. secret:
  37. secretName: federation-controller-manager-kubeconfig

通过 ConfigMaps 配置放置策略

外部策略引擎将发现在 Federation API 服务器的 kube-federation-scheduling-policy 命名空间中创建的放置策略。

如果命名空间尚不存在,请创建它:

  1. kubectl --context=federation create namespace kube-federation-scheduling-policy

配置一个示例策略来测试外部策略引擎:

  1. # OPA supports a high-level declarative language named Rego for authoring and
  2. # enforcing policies. For more information on Rego, visit
  3. # http://openpolicyagent.org.
  4. # Rego policies are namespaced by the "package" directive.
  5. package kubernetes.placement
  6. # Imports provide aliases for data inside the policy engine. In this case, the
  7. # policy simply refers to "clusters" below.
  8. import data.kubernetes.clusters
  9. # The "annotations" rule generates a JSON object containing the key
  10. # "federation.kubernetes.io/replica-set-preferences" mapped to <preferences>.
  11. # The preferences values is generated dynamically by OPA when it evaluates the
  12. # rule.
  13. #
  14. # The SchedulingPolicy Admission Controller running inside the Federation API
  15. # server will merge these annotations into incoming Federated resources. By
  16. # setting replica-set-preferences, we can control the placement of Federated
  17. # ReplicaSets.
  18. #
  19. # Rules are defined to generate JSON values (booleans, strings, objects, etc.)
  20. # When OPA evaluates a rule, it generates a value IF all of the expressions in
  21. # the body evaluate successfully. All rules can be understood intuitively as
  22. # <head> if <body> where <body> is true if <expr-1> AND <expr-2> AND ...
  23. # <expr-N> is true (for some set of data.)
  24. annotations["federation.kubernetes.io/replica-set-preferences"] = preferences {
  25. input.kind = "ReplicaSet"
  26. value = {"clusters": cluster_map, "rebalance": true}
  27. json.marshal(value, preferences)
  28. }
  29. # This "annotations" rule generates a value for the "federation.alpha.kubernetes.io/cluster-selector"
  30. # annotation.
  31. #
  32. # In English, the policy asserts that resources in the "production" namespace
  33. # that are not annotated with "criticality=low" MUST be placed on clusters
  34. # labelled with "on-premises=true".
  35. annotations["federation.alpha.kubernetes.io/cluster-selector"] = selector {
  36. input.metadata.namespace = "production"
  37. not input.metadata.annotations.criticality = "low"
  38. json.marshal([{
  39. "operator": "=",
  40. "key": "on-premises",
  41. "values": "[true]",
  42. }], selector)
  43. }
  44. # Generates a set of cluster names that satisfy the incoming Federated
  45. # ReplicaSet's requirements. In this case, just PCI compliance.
  46. replica_set_clusters[cluster_name] {
  47. clusters[cluster_name]
  48. not insufficient_pci[cluster_name]
  49. }
  50. # Generates a set of clusters that must not be used for Federated ReplicaSets
  51. # that request PCI compliance.
  52. insufficient_pci[cluster_name] {
  53. clusters[cluster_name]
  54. input.metadata.annotations["requires-pci"] = "true"
  55. not pci_clusters[cluster_name]
  56. }
  57. # Generates a set of clusters that are PCI certified. In this case, we assume
  58. # clusters are annotated to indicate if they have passed PCI compliance audits.
  59. pci_clusters[cluster_name] {
  60. clusters[cluster_name].metadata.annotations["pci-certified"] = "true"
  61. }
  62. # Helper rule to generate a mapping of desired clusters to weights. In this
  63. # case, weights are static.
  64. cluster_map[cluster_name] = {"weight": 1} {
  65. replica_set_clusters[cluster_name]
  66. }

下面显示的是创建示例策略的命令:

  1. kubectl --context=federation -n kube-federation-scheduling-policy create configmap scheduling-policy --from-file=policy.rego

这个示例策略说明了一些关键思想:

  • 位置策略可以引用联邦资源中的任何字段。
  • 放置策略可以利用外部上下文(例如,集群元数据)来做出决策。
  • 管理策略可以集中管理。
  • 策略可以定义简单的接口(例如 requirements -pci 注解),以避免在清单中重复逻辑。

测试放置政策

注释其中一个集群以表明它是经过 PCI 认证的。

  1. kubectl --context=federation annotate clusters cluster-name-1 pci-certified=true

部署联邦副本来测试放置策略。

federation/replicaset-example-policy.yaml 在联邦中设置放置策略 - 图4
  1. apiVersion: apps/v1
  2. kind: ReplicaSet
  3. metadata:
  4. labels:
  5. app: nginx-pci
  6. name: nginx-pci
  7. annotations:
  8. requires-pci: true
  9. spec:
  10. replicas: 3
  11. selector:
  12. matchLabels:
  13. app: nginx-pci
  14. template:
  15. metadata:
  16. labels:
  17. app: nginx-pci
  18. spec:
  19. containers:
  20. - image: nginx
  21. name: nginx-pci

下面显示的命令用于部署与策略匹配的副本集。

  1. kubectl --context=federation create -f replicaset-example-policy.yaml

检查副本集以确认已应用适当的注解:

  1. kubectl --context=federation get rs nginx-pci -o jsonpath='{.metadata.annotations}'