Istio Webhook Management [Experimental]

The following information describes an experimental feature, which is intendedfor evaluation purposes only.

Istio has two webhooks: Galley and the sidecar injector. By default,these webhooks manage their own configurations. From asecurity perspective, this default behavior is not recommended because a compromised webhook could then conductprivilege escalation attacks.

This task shows how to use the new istioctl x post-install webhook command tosecurely manage the configurations of the webhooks.

Getting started

  1. $ cat <<EOF > ./istio.yaml
  2. apiVersion: install.istio.io/v1alpha2
  3. kind: IstioControlPlane
  4. spec:
  5.   values:
  6.     global:
  7.       operatorManageWebhooks: true
  8.       certificates:
  9.         - secretName: dns.istio-galley-service-account
  10.           dnsNames: [istio-galley.istio-system.svc, istio-galley.istio-system]
  11.         - secretName: dns.istio-sidecar-injector-service-account
  12.           dnsNames: [istio-sidecar-injector.istio-system.svc, istio-sidecar-injector.istio-system]
  13. EOF
  14. $ istioctl manifest apply -f ./istio.yaml
  • Install jq for JSON parsing.

Check webhook certificates

To display the DNS names in the webhook certificates of Galley and the sidecar injector, you need to get the secretfrom Kubernetes, parse it, decode it, and view the text output with the following commands:

  1. $ kubectl get secret dns.istio-galley-service-account -n istio-system -o json | jq -r '.data["cert-chain.pem"]' | base64 --decode | openssl x509 -in - -text -noout
  2. $ kubectl get secret dns.istio-sidecar-injector-service-account -n istio-system -o json | jq -r '.data["cert-chain.pem"]' | base64 --decode | openssl x509 -in - -text -noout

The output from the above commands should include the DNS names of Galley and the sidecar injector, respectively:

  1. X509v3 Subject Alternative Name:
  2.   DNS:istio-galley.istio-system.svc, DNS:istio-galley.istio-system
  1. X509v3 Subject Alternative Name:
  2.   DNS:istio-sidecar-injector.istio-system.svc, DNS:istio-sidecar-injector.istio-system

Enable webhook configurations

  • To generate the MutatingWebhookConfiguration and ValidatingWebhookConfiguration configuration files, run the followingcommand.
  1. $ istioctl manifest generate > istio.yaml
  • Open the istio.yaml configuration file, search for kind: MutatingWebhookConfiguration and savethe MutatingWebhookConfiguration of the sidecar injector to sidecar-injector-webhook.yaml. The followingis a MutatingWebhookConfiguration in an example istio.yaml.
  1. apiVersion: admissionregistration.k8s.io/v1beta1
  2. kind: MutatingWebhookConfiguration
  3. metadata:
  4.   name: istio-sidecar-injector
  5.   labels:
  6.     app: sidecarInjectorWebhook
  7.     release: istio
  8. webhooks:
  9.   - name: sidecar-injector.istio.io
  10.     clientConfig:
  11.       service:
  12.         name: istio-sidecar-injector
  13.         namespace: istio-system
  14.         path: "/inject"
  15.       caBundle: ""
  16.     rules:
  17.       - operations: [ "CREATE" ]
  18.         apiGroups: [""]
  19.         apiVersions: ["v1"]
  20.         resources: ["pods"]
  21.     failurePolicy: Fail
  22.     namespaceSelector:
  23.       matchLabels:
  24.         istio-injection: enabled
  • Open the istio.yaml configuration file, search for kind: ValidatingWebhookConfiguration and savethe ValidatingWebhookConfiguration of Galley to galley-webhook.yaml. The followingis a ValidatingWebhookConfiguration in an example istio.yaml (onlya part of the configuration is shown to save space).
  1. apiVersion: admissionregistration.k8s.io/v1beta1
  2. kind: ValidatingWebhookConfiguration
  3. metadata:
  4.   name: istio-galley
  5.   labels:
  6.     app: galley
  7.     release: istio
  8.     istio: galley
  9. webhooks:
  10.   - name: pilot.validation.istio.io
  11.     clientConfig:
  12.       service:
  13.         name: istio-galley
  14.         namespace: istio-system
  15.         path: "/admitpilot"
  16.       caBundle: ""
  17.     rules:
  18.       - operations:
  19.         - CREATE
  20.         - UPDATE
  21.         apiGroups:
  22.         - config.istio.io
  23.         ... SKIPPED
  24.     failurePolicy: Fail
  25.     sideEffects: None
  • Verify that there are no existing webhook configurations for Galley and the sidecar injector.The output of the following two commands should not contain any configurations forGalley and the sidecar injector.
  1. $ kubectl get mutatingwebhookconfiguration
  2. $ kubectl get validatingwebhookconfiguration

If there are existing webhook configurations (e.g., from a previous Istio deployment) forGalley and the sidecar injector, delete them using the following commands. Before runningthese commands, replace the webhook configuration names in the commands with theactual webhook configuration names of Galley and the sidecar injector in your cluster.

  1. $ kubectl delete mutatingwebhookconfiguration SIDECAR-INJECTOR-WEBHOOK-CONFIGURATION-NAME
  2. $ kubectl delete validatingwebhookconfiguration GALLEY-WEBHOOK-CONFIGURATION-NAME
  • Use istioctl to enable the webhook configurations:
  1. $ istioctl experimental post-install webhook enable --webhook-secret dns.istio-galley-service-account \
  2.     --namespace istio-system --validation-path galley-webhook.yaml \
  3.     --injection-path sidecar-injector-webhook.yaml
  • To check that the sidecar injector webhook is working, verify that the webhook injects asidecar container into an example pod with the following commands:
  1. $ kubectl create namespace test-injection
  2. $ kubectl label namespaces test-injection istio-injection=enabled
  3. $ kubectl run --generator=run-pod/v1 --image=nginx nginx-app --port=80 -n test-injection
  4. $ kubectl get pod -n test-injection

The output from the get pod command should show the following. The 2/2 value means thatthe webhook injected a sidecar into the example pod:

  1. NAME        READY   STATUS    RESTARTS   AGE
  2. nginx-app   2/2     Running   0          10s
  • Check that the validation webhook is working:
  1. $ kubectl create namespace test-validation
  2. $ kubectl apply -n test-validation -f - <<EOF
  3. apiVersion: networking.istio.io/v1alpha3
  4. kind: Gateway
  5. metadata:
  6.   name: invalid-gateway
  7. spec:
  8.   selector:
  9.     # DO NOT CHANGE THESE LABELS
  10.     # The ingressgateway is defined in install/kubernetes/helm/istio/values.yaml
  11.     # with these labels
  12.     istio: ingressgateway
  13. EOF

The output from the gateway creation command should show the following output. The errorin the output indicates that the validation webhook checked the gateway’s configuration YAML file:

  1. Error from server: error when creating "invalid-gateway.yaml": admission webhook "pilot.validation.istio.io" denied the request: configuration is invalid: gateway must have at least one server

Show webhook configurations

  • If you named the sidecar injector’s configuration istio-sidecar-injector andnamed Galley’s configuration istio-galley-istio-system, use the following commandto show the configurations of these two webhooks:
  1. $ istioctl experimental post-install webhook status --validation-config=istio-galley-istio-system  --injection-config=istio-sidecar-injector
  • If you named the sidecar injector’s configuration istio-sidecar-injector,use the following command to show the configuration of the sidecar injector:
  1. $ istioctl experimental post-install webhook status --validation=false --injection-config=istio-sidecar-injector
  • If you named Galley’s configuration istio-galley-istio-system, show Galley’s configuration with the following command:
  1. $ istioctl experimental post-install webhook status --injection=false --validation-config=istio-galley-istio-system

Disable webhook configurations

  • If you named the sidecar injector’s configuration istio-sidecar-injector andnamed Galley’s configuration istio-galley-istio-system, use the following commandto disable the configurations of these two webhooks:
  1. $ istioctl experimental post-install webhook disable --validation-config=istio-galley-istio-system  --injection-config=istio-sidecar-injector
  • If you named the sidecar injector’s configuration istio-sidecar-injector,disable the webhook with the following command:
  1. $ istioctl experimental post-install webhook disable --validation=false --injection-config=istio-sidecar-injector
  • If you named Galleys’s configuration istio-galley-istio-system, disable the webhook with the following command:
  1. $ istioctl experimental post-install webhook disable --injection=false --validation-config=istio-galley-istio-system

Cleanup

You can run the following command to delete the resources created in this tutorial.

  1. $ kubectl delete ns test-injection test-validation
  2. $ kubectl delete -f galley-webhook.yaml
  3. $ kubectl delete -f sidecar-injector-webhook.yaml

See also

Secure Webhook Management

A more secure way to manage Istio webhooks.

DNS Certificate Management

Provision and manage DNS certificates in Istio.

Introducing the Istio v1beta1 Authorization Policy

Introduction, motivation and design principles for the Istio v1beta1 Authorization Policy.

Multi-Mesh Deployments for Isolation and Boundary Protection

Deploy environments that require isolation into separate meshes and enable inter-mesh communication by mesh federation.

App Identity and Access Adapter

Using Istio to secure multi-cloud Kubernetes applications with zero code changes.

Change in Secret Discovery Service in Istio 1.3

Taking advantage of Kubernetes trustworthy JWTs to issue certificates for workload instances more securely.