Working with Gatekeeper(OPA)

Gatekeeper , is a customizable admission webhook for Kubernetes that enforces policies executed by the Open Policy Agent (OPA), a policy engine for Cloud Native environments hosted by Cloud Native Computing Foundation.

This document demonstrates how to use the Gatekeeper to manage OPA policies.

Prerequisites

Start up Karmada clusters

You just need to clone Karmada repo, and run the following script in the Karmada directory.

  1. hack/local-up-karmada.sh

Gatekeeper Installations

In this case, you will use Gatekeeper v3.7.2. Related deployment files are from here.

Install Gatekeeper APIs on Karmada

  1. Create resource objects for Gatekeeper in karmada controller plane, the content is as follows.

    1. kubectl config use-context karmada-apiserver

    Deploy namespace: https://github.com/open-policy-agent/gatekeeper/blob/release-3.7/deploy/gatekeeper.yaml#L1-L9

    Deploy Gatekeeper CRDs: https://github.com/open-policy-agent/gatekeeper/blob/release-3.7/deploy/gatekeeper.yaml#L27-L1999

    Deploy Gatekeeper secrets: https://github.com/open-policy-agent/gatekeeper/blob/release-3.7/deploy/gatekeeper.yaml#L2261-L2267

    Deploy webhook config:

    1. apiVersion: admissionregistration.k8s.io/v1
    2. kind: MutatingWebhookConfiguration
    3. metadata:
    4.   labels:
    5.     gatekeeper.sh/system: "yes"
    6.   name: gatekeeper-mutating-webhook-configuration
    7. webhooks:
    8.   - admissionReviewVersions:
    9.       - v1
    10.       - v1beta1
    11.     clientConfig:
    12.       #Change the clientconfig from service type to url type cause webhook config and service are not in the same cluster.
    13.       url: https://gatekeeper-webhook-service.gatekeeper-system.svc:443/v1/mutate
    14.     failurePolicy: Ignore
    15.     matchPolicy: Exact
    16.     name: mutation.gatekeeper.sh
    17.     namespaceSelector:
    18.       matchExpressions:
    19.         - key: admission.gatekeeper.sh/ignore
    20.           operator: DoesNotExist
    21.     rules:
    22.       - apiGroups:
    23.           - '*'
    24.         apiVersions:
    25.           - '*'
    26.         operations:
    27.           - CREATE
    28.           - UPDATE
    29.         resources:
    30.           - '*'
    31.     sideEffects: None
    32.     timeoutSeconds: 1
    33. ---
    34. apiVersion: admissionregistration.k8s.io/v1
    35. kind: ValidatingWebhookConfiguration
    36. metadata:
    37.   labels:
    38.     gatekeeper.sh/system: "yes"
    39.   name: gatekeeper-validating-webhook-configuration
    40. webhooks:
    41.   - admissionReviewVersions:
    42.       - v1
    43.       - v1beta1
    44.     clientConfig:
    45.       #Change the clientconfig from service type to url type cause webhook config and service are not in the same cluster.
    46.       url: https://gatekeeper-webhook-service.gatekeeper-system.svc:443/v1/admit
    47.     failurePolicy: Ignore
    48.     matchPolicy: Exact
    49.     name: validation.gatekeeper.sh
    50.     namespaceSelector:
    51.       matchExpressions:
    52.         - key: admission.gatekeeper.sh/ignore
    53.           operator: DoesNotExist
    54.     rules:
    55.       - apiGroups:
    56.           - '*'
    57.         apiVersions:
    58.           - '*'
    59.         operations:
    60.           - CREATE
    61.           - UPDATE
    62.         resources:
    63.           - '*'
    64.     sideEffects: None
    65.     timeoutSeconds: 3
    66.   - admissionReviewVersions:
    67.       - v1
    68.       - v1beta1
    69.     clientConfig:
    70.       #Change the clientconfig from service type to url type cause webhook config and service are not in the same cluster.
    71.       url: https://gatekeeper-webhook-service.gatekeeper-system.svc:443/v1/admitlabel
    72.     failurePolicy: Fail
    73.     matchPolicy: Exact
    74.     name: check-ignore-label.gatekeeper.sh
    75.     rules:
    76.       - apiGroups:
    77.           - ""
    78.         apiVersions:
    79.           - '*'
    80.         operations:
    81.           - CREATE
    82.           - UPDATE
    83.         resources:
    84.           - namespaces
    85.     sideEffects: None
    86.     timeoutSeconds: 3

    You need to change the clientconfig from service type to url type for multi-cluster deployment.

    Also, you need to deploy a dummy pod in gatekeeper-system namespace in karmada-apiserver context, because when Gatekeeper generates a policy template CRD, a status object is generated to monitor the status of the policy template, and the status object is bound by the controller Pod through the OwnerReference. Therefore, when the CRD and the controller are not in the same cluster, a dummy Pod needs to be used instead of the controller. The Pod enables the status object to be successfully generated.

    For example:

    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4.   name: dummy-pod
    5.   namespace: gatekeeper-system
    6. spec:
    7.   containers:
    8.   - name: dummy-pod
    9.     image: nginx:latest
    10.     imagePullPolicy: Always

Install GateKeeper components on host cluster

  1. kubectl config use-context karmada-host

Deploy namespace: https://github.com/open-policy-agent/gatekeeper/blob/release-3.7/deploy/gatekeeper.yaml#L1-L9

Deploy RBAC resources for deployment: https://github.com/open-policy-agent/gatekeeper/blob/release-3.7/deploy/gatekeeper.yaml#L1999-L2375

Deploy Gatekeeper controllers and secret as kubeconfig:

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   labels:
  5.     control-plane: audit-controller
  6.     gatekeeper.sh/operation: audit
  7.     gatekeeper.sh/system: "yes"
  8.   name: gatekeeper-audit
  9.   namespace: gatekeeper-system
  10. spec:
  11.   replicas: 1
  12.   selector:
  13.     matchLabels:
  14.       control-plane: audit-controller
  15.       gatekeeper.sh/operation: audit
  16.       gatekeeper.sh/system: "yes"
  17.   template:
  18.     metadata:
  19.       annotations:
  20.         container.seccomp.security.alpha.kubernetes.io/manager: runtime/default
  21.       labels:
  22.         control-plane: audit-controller
  23.         gatekeeper.sh/operation: audit
  24.         gatekeeper.sh/system: "yes"
  25.     spec:
  26.       automountServiceAccountToken: true
  27.       containers:
  28.         - args:
  29.             - --operation=audit
  30.             - --operation=status
  31.             - --operation=mutation-status
  32.             - --logtostderr
  33.             - --disable-opa-builtin={http.send}
  34.             - --kubeconfig=/etc/kubeconfig
  35.           command:
  36.             - /manager
  37.           env:
  38.             - name: POD_NAMESPACE
  39.               valueFrom:
  40.                 fieldRef:
  41.                   apiVersion: v1
  42.                   fieldPath: metadata.namespace
  43.             - name: POD_NAME
  44.               value: {{POD_NAME}}
  45.           image: openpolicyagent/gatekeeper:v3.7.2
  46.           imagePullPolicy: Always
  47.           livenessProbe:
  48.             httpGet:
  49.               path: /healthz
  50.               port: 9090
  51.           name: manager
  52.           ports:
  53.             - containerPort: 8888
  54.               name: metrics
  55.               protocol: TCP
  56.             - containerPort: 9090
  57.               name: healthz
  58.               protocol: TCP
  59.           readinessProbe:
  60.             httpGet:
  61.               path: /readyz
  62.               port: 9090
  63.           resources:
  64.             limits:
  65.               cpu: 1000m
  66.               memory: 512Mi
  67.             requests:
  68.               cpu: 100m
  69.               memory: 256Mi
  70.           securityContext:
  71.             allowPrivilegeEscalation: false
  72.             capabilities:
  73.               drop:
  74.                 - all
  75.             readOnlyRootFilesystem: true
  76.             runAsGroup: 999
  77.             runAsNonRoot: true
  78.             runAsUser: 1000
  79.           volumeMounts:
  80.             - mountPath: /tmp/audit
  81.               name: tmp-volume
  82.             - mountPath: /etc/kubeconfig
  83.               name: kubeconfig
  84.               subPath: kubeconfig
  85.       nodeSelector:
  86.         kubernetes.io/os: linux
  87.       priorityClassName: system-cluster-critical
  88.       serviceAccountName: gatekeeper-admin
  89.       terminationGracePeriodSeconds: 60
  90.       volumes:
  91.         - emptyDir: {}
  92.           name: tmp-volume
  93.         - name: kubeconfig
  94.           secret:
  95.             defaultMode: 420
  96.             secretName: kubeconfig
  97. ---
  98. apiVersion: apps/v1
  99. kind: Deployment
  100. metadata:
  101.   labels:
  102.     control-plane: controller-manager
  103.     gatekeeper.sh/operation: webhook
  104.     gatekeeper.sh/system: "yes"
  105.   name: gatekeeper-controller-manager
  106.   namespace: gatekeeper-system
  107. spec:
  108.   replicas: 3
  109.   selector:
  110.     matchLabels:
  111.       control-plane: controller-manager
  112.       gatekeeper.sh/operation: webhook
  113.       gatekeeper.sh/system: "yes"
  114.   template:
  115.     metadata:
  116.       annotations:
  117.         container.seccomp.security.alpha.kubernetes.io/manager: runtime/default
  118.       labels:
  119.         control-plane: controller-manager
  120.         gatekeeper.sh/operation: webhook
  121.         gatekeeper.sh/system: "yes"
  122.     spec:
  123.       affinity:
  124.         podAntiAffinity:
  125.           preferredDuringSchedulingIgnoredDuringExecution:
  126.             - podAffinityTerm:
  127.                 labelSelector:
  128.                   matchExpressions:
  129.                     - key: gatekeeper.sh/operation
  130.                       operator: In
  131.                       values:
  132.                         - webhook
  133.                 topologyKey: kubernetes.io/hostname
  134.               weight: 100
  135.       automountServiceAccountToken: true
  136.       containers:
  137.         - args:
  138.             - --port=8443
  139.             - --logtostderr
  140.             - --exempt-namespace=gatekeeper-system
  141.             - --operation=webhook
  142.             - --operation=mutation-webhook
  143.             - --disable-opa-builtin={http.send}
  144.             - --kubeconfig=/etc/kubeconfig
  145.           command:
  146.             - /manager
  147.           env:
  148.             - name: POD_NAMESPACE
  149.               valueFrom:
  150.                 fieldRef:
  151.                   apiVersion: v1
  152.                   fieldPath: metadata.namespace
  153.             - name: POD_NAME
  154.               value: {{POD_NAME}}
  155.           image: openpolicyagent/gatekeeper:v3.7.2
  156.           imagePullPolicy: Always
  157.           livenessProbe:
  158.             httpGet:
  159.               path: /healthz
  160.               port: 9090
  161.           name: manager
  162.           ports:
  163.             - containerPort: 8443
  164.               name: webhook-server
  165.               protocol: TCP
  166.             - containerPort: 8888
  167.               name: metrics
  168.               protocol: TCP
  169.             - containerPort: 9090
  170.               name: healthz
  171.               protocol: TCP
  172.           readinessProbe:
  173.             httpGet:
  174.               path: /readyz
  175.               port: 9090
  176.           resources:
  177.             limits:
  178.               cpu: 1000m
  179.               memory: 512Mi
  180.             requests:
  181.               cpu: 100m
  182.               memory: 256Mi
  183.           securityContext:
  184.             allowPrivilegeEscalation: false
  185.             capabilities:
  186.               drop:
  187.                 - all
  188.             readOnlyRootFilesystem: true
  189.             runAsGroup: 999
  190.             runAsNonRoot: true
  191.             runAsUser: 1000
  192.           volumeMounts:
  193.             - mountPath: /certs
  194.               name: cert
  195.               readOnly: true
  196.             - mountPath: /etc/kubeconfig
  197.               name: kubeconfig
  198.               subPath: kubeconfig
  199.       nodeSelector:
  200.         kubernetes.io/os: linux
  201.       priorityClassName: system-cluster-critical
  202.       serviceAccountName: gatekeeper-admin
  203.       terminationGracePeriodSeconds: 60
  204.       volumes:
  205.         - name: cert
  206.           secret:
  207.             defaultMode: 420
  208.             secretName: gatekeeper-webhook-server-cert
  209.         - name: kubeconfig
  210.           secret:
  211.             defaultMode: 420
  212.             secretName: kubeconfig
  213. ---
  214. apiVersion: policy/v1beta1
  215. kind: PodDisruptionBudget
  216. metadata:
  217.   labels:
  218.     gatekeeper.sh/system: "yes"
  219.   name: gatekeeper-controller-manager
  220.   namespace: gatekeeper-system
  221. spec:
  222.   minAvailable: 1
  223.   selector:
  224.     matchLabels:
  225.       control-plane: controller-manager
  226.       gatekeeper.sh/operation: webhook
  227.       gatekeeper.sh/system: "yes"
  228. ---
  229. apiVersion: v1
  230. stringData:
  231.   kubeconfig: |-
  232.     apiVersion: v1
  233.     clusters:
  234.     - cluster:
  235.         certificate-authority-data: {{ca_crt}}
  236.         server: https://karmada-apiserver.karmada-system.svc.cluster.local:5443
  237.       name: kind-karmada
  238.     contexts:
  239.     - context:
  240.         cluster: kind-karmada
  241.         user: kind-karmada
  242.       name: karmada
  243.     current-context: karmada
  244.     kind: Config
  245.     preferences: {}
  246.     users:
  247.     - name: kind-karmada
  248.       user:
  249.         client-certificate-data: {{client_cer}}
  250.         client-key-data: {{client_key}}
  251. kind: Secret
  252. metadata:
  253.   name: kubeconfig
  254.   namespace: gatekeeper-system

You need to fill in the dummy pod created in step 1 to {{ POD_NAME }} and fill in the secret which represents kubeconfig pointing to karmada-apiserver.

Deploy ResourceQuota: https://github.com/open-policy-agent/gatekeeper/blob/release-3.7/deploy/gatekeeper.yaml#L10-L26

Extra steps

After all, we need to copy the secret gatekeeper-webhook-server-cert in karmada-apiserver context to that in karmada-host context to keep secrets stored in etcd and volumes mounted in controller the same.

Run demo

Create k8srequiredlabels template

  1. apiVersion: templates.gatekeeper.sh/v1
  2. kind: ConstraintTemplate
  3. metadata:
  4.   name: k8srequiredlabels
  5. spec:
  6.   crd:
  7.     spec:
  8.       names:
  9.         kind: K8sRequiredLabels
  10.       validation:
  11.         openAPIV3Schema:  
  12.           type: object
  13.           description: Describe K8sRequiredLabels crd parameters
  14.           properties:
  15.             labels:
  16.               type: array
  17.               items:
  18.                 type: string
  19.                 description: A label string
  20.   targets:
  21.     - target: admission.k8s.gatekeeper.sh
  22.       rego: |
  23.         package k8srequiredlabels
  25.         violation[{"msg": msg, "details": {"missing_labels": missing}}] {
  26.           provided := {label | input.review.object.metadata.labels[label]}
  27.           required := {label | label := input.parameters.labels[_]}
  28.           missing := required - provided
  29.           count(missing) > 0
  30.           msg := sprintf("you must provide labels: %v", [missing])
  31.         }

Create k8srequiredlabels constraint

  1. apiVersion: constraints.gatekeeper.sh/v1beta1
  2. kind: K8sRequiredLabels
  3. metadata:
  4.   name: ns-must-have-gk
  5. spec:
  6.   match:
  7.     kinds:
  8.       - apiGroups: [""]
  9.         kinds: ["Namespace"]  
  10.   parameters:
  11.     labels: ["gatekeepers"]

Create a bad namespace

  1. $ kubectl create ns test
  2. Error from server ([ns-must-have-gk] you must provide labels: {"gatekeepers"}): admission webhook "validation.gatekeeper.sh" denied the request: [ns-must-have-gk] you must provide labels: {"gatekeepers"}

Reference