Configuration Validation Problems

Seemingly valid configuration is rejected

Use istioctl validate -f and istioctl analyze for more insight into why the configuration is rejected. Use an istioctl CLI with a similar version to the control plane version.

The most commonly reported problems with configuration are YAML indentation and array notation (-) mistakes.

Manually verify your configuration is correct, cross-referencing Istio API reference when necessary.

Invalid configuration is accepted

Verify the istiod-istio-system validationwebhookconfiguration exists and is correct. The apiVersion, apiGroup, and resource of the invalid configuration should be listed in one of the two webhooks entries.

  1. $ kubectl get validatingwebhookconfiguration istiod-istio-system -o yaml
  2. apiVersion: admissionregistration.k8s.io/v1
  3. kind: ValidatingWebhookConfiguration
  4. metadata:
  5.   creationTimestamp: "2020-01-24T19:53:03Z"
  6.   generation: 1
  7.   labels:
  8.     app: istiod
  9.     istio: istiod
  10.     release: istio
  11.   name: istiod-istio-system
  12.   ownerReferences:
  13.   - apiVersion: rbac.authorization.k8s.io/v1
  14.     blockOwnerDeletion: true
  15.     controller: true
  16.     kind: ClusterRole
  17.     name: istiod-istio-system
  18.     uid: c3d24917-c2da-49ad-add3-c91c14608a45
  19.   resourceVersion: "36649"
  20.   selfLink: /apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations/istiod-istio-system
  21.   uid: 043e39d9-377a-4a67-a7cf-7ae4cb3c562c
  22. webhooks:
  23. - admissionReviewVersions:
  24.   - v1beta1
  25.   clientConfig:
  26.     # caBundle should be non-empty. This is periodically (re)patched
  27.     # every second by the webhook service using the ca-cert
  28.     # from the mounted service account secret.
  29.     caBundle: LS0t...
  30.     service:
  31.       # service corresponds to the Kubernetes service that implements the
  32.       # webhook, e.g. istio-galley.istio-system.svc:443
  33.       name: istiod
  34.       namespace: istio-system
  35.       path: /validate
  36.       port: 443
  37.   failurePolicy: Fail
  38.   matchPolicy: Exact
  39.   name: validation.istio.io
  40.   namespaceSelector: {}
  41.   objectSelector: {}
  42.   rules:
  43.   - apiGroups:
  44.     - config.istio.io
  45.     - rbac.istio.io
  46.     - security.istio.io
  47.     - authentication.istio.io
  48.     - networking.istio.io
  49.     apiVersions:
  50.     - '*'
  51.     operations:
  52.     - CREATE
  53.     - UPDATE
  54.     resources:
  55.     - '*'
  56.     scope: '*'
  57.   sideEffects: None
  58.   timeoutSeconds: 30

If the validatingwebhookconfiguration doesn’t exist, verify the istio-validation configmap exists. Istio uses the data from this configmap to create and update the validatingwebhookconfiguration.

  1. $ kubectl -n istio-system get configmap istio-validation -o jsonpath='{.data}'
  2. map[config:apiVersion: admissionregistration.k8s.io/v1beta1
  3. kind: ValidatingWebhookConfiguration
  4. metadata:
  5.   name: istiod-istio-system
  6.   namespace: istio-system
  7.   labels:
  8.     app: istiod
  9.     release: istio
  10.     istio: istiod
  11. webhooks:
  12.   - name: validation.istio.io
  13.     clientConfig:
  14.       service:
  15.         name: istiod
  16.         namespace: istio-system
  17.         path: "/validate"
  18.         port: 443
  19.       caBundle: ""
  20.     rules:
  21.       - operations:
  22.         - CREATE
  23.         - UPDATE
  24.         apiGroups:
  25.         - config.istio.io
  26.         - rbac.istio.io
  27.         - security.istio.io
  28.         - authentication.istio.io
  29.         - networking.istio.io
  30.         apiVersions:
  31.         - "*"
  32.         resources:
  33.         - "*"
  34.     failurePolicy: Fail
  35.     sideEffects: None]
  36.       (... snip ...)

If the webhook array in istio-validation is empty, verify the global.configValidation installation options are set.

The validation configuration is fail-close. If configuration exists and is scoped properly, the webhook will be invoked. A missing caBundle, bad certificate, or network connectivity problem will produce an error message when the resource is created/updated. If you don’t see any error message and the webhook wasn’t invoked and the webhook configuration is valid, your cluster is misconfigured.

Creating configuration fails with x509 certificate errors

x509: certificate signed by unknown authority related errors are typically caused by an empty caBundle in the webhook configuration. Verify that it is not empty (see verify webhook configuration). Istio consciously reconciles webhook configuration used the istio-validation configmap and root certificate.

  1. Verify the istiod pod(s) are running:

    1. $  kubectl -n istio-system get pod -lapp=istiod
    2. NAME                            READY     STATUS    RESTARTS   AGE
    3. istiod-5dbbbdb746-d676g   1/1       Running   0          2d

  2. Check the pod logs for errors. Failing to patch the caBundle should print an error.

    1. $ for pod in $(kubectl -n istio-system get pod -lapp=istiod -o jsonpath='{.items[*].metadata.name}'); do \
    2.     kubectl -n istio-system logs ${pod} \
    3. done

  3. If the patching failed, verify the RBAC configuration for Istiod:

    1. $ kubectl get clusterrole istiod-istio-system -o yaml
    2. apiVersion: rbac.authorization.k8s.io/v1
    3. kind: ClusterRole
    4.   name: istiod-istio-system
    5. rules:
    6. - apiGroups:
    7.   - admissionregistration.k8s.io
    8.   resources:
    9.   - validatingwebhookconfigurations
    10.   verbs:
    11.   - '*'

    Istio needs validatingwebhookconfigurations write access to create and update the validatingwebhookconfiguration.

Creating configuration fails with no such hosts or no endpoints available errors

Validation is fail-close. If the istiod pod is not ready, configuration cannot be created and updated. In such cases you’ll see an error about no endpoints available.

Verify the istiod pod(s) are running and endpoints are ready.

  1. $  kubectl -n istio-system get pod -lapp=istiod
  2. NAME                            READY     STATUS    RESTARTS   AGE
  3. istiod-5dbbbdb746-d676g   1/1       Running   0          2d
  1. $ kubectl -n istio-system get endpoints istiod
  2. NAME           ENDPOINTS                          AGE
  3. istiod         10.48.6.108:15014,10.48.6.108:443   3d

If the pods or endpoints aren’t ready, check the pod logs and status for any indication about why the webhook pod is failing to start and serve traffic.

  1. $ for pod in $(kubectl -n istio-system get pod -lapp=istiod -o jsonpath='{.items[*].metadata.name}'); do \
  2.     kubectl -n istio-system logs ${pod} \
  3. done
  1. $ for pod in $(kubectl -n istio-system get pod -lapp=istiod -o name); do \
  2.     kubectl -n istio-system describe ${pod} \
  3. done