Sidecar Injection Problems

The result of sidecar injection was not what I expected

This includes an injected sidecar when it wasn’t expected and a lack of injected sidecar when it was.

  1. Ensure your pod is not in the kube-system or kube-public namespace. Automatic sidecar injection will be ignored for pods in these namespaces.

  2. Ensure your pod does not have hostNetwork: true in its pod spec. Automatic sidecar injection will be ignored for pods that are on the host network.

    The sidecar model assumes that the iptables changes required for Envoy to intercept traffic are within the pod. For pods on the host network this assumption is violated, and this can lead to routing failures at the host level.

  3. Check the webhook’s namespaceSelector to determine whether the webhook is scoped to opt-in or opt-out for the target namespace.

    The namespaceSelector for opt-in will look like the following:

    1. $ kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml | grep "namespaceSelector:" -A5
    2.   namespaceSelector:
    3.     matchLabels:
    4.       istio-injection: enabled
    5.   rules:
    6.   - apiGroups:
    7.     - ""

    The injection webhook will be invoked for pods created in namespaces with the istio-injection=enabled label.

    1. $ kubectl get namespace -L istio-injection
    2. NAME           STATUS    AGE       ISTIO-INJECTION
    3. default        Active    18d       enabled
    4. istio-system   Active    3d
    5. kube-public    Active    18d
    6. kube-system    Active    18d

    The namespaceSelector for opt-out will look like the following:

    1. $ kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml | grep "namespaceSelector:" -A5
    2.   namespaceSelector:
    3.     matchExpressions:
    4.     - key: istio-injection
    5.       operator: NotIn
    6.       values:
    7.       - disabled
    8.   rules:
    9.   - apiGroups:
    10.     - ""

    The injection webhook will be invoked for pods created in namespaces without the istio-injection=disabled label.

    1. $ kubectl get namespace -L istio-injection
    2. NAME           STATUS    AGE       ISTIO-INJECTION
    3. default        Active    18d
    4. istio-system   Active    3d        disabled
    5. kube-public    Active    18d       disabled
    6. kube-system    Active    18d       disabled

    Verify the application pod’s namespace is labeled properly and (re) label accordingly, e.g.

    1. $ kubectl label namespace istio-system istio-injection=disabled --overwrite

    (repeat for all namespaces in which the injection webhook should be invoked for new pods)

    1. $ kubectl label namespace default istio-injection=enabled --overwrite

  4. Check default policy

    Check the default injection policy in the istio-sidecar-injector configmap.

    1. $ kubectl -n istio-system get configmap istio-sidecar-injector -o jsonpath='{.data.config}' | grep policy:
    2. policy: enabled

    Allowed policy values are disabled and enabled. The default policy only applies if the webhook’s namespaceSelector matches the target namespace. Unrecognized policy causes injection to be disabled completely.

  5. Check the per-pod override annotation

    The default policy can be overridden with the sidecar.istio.io/inject label in the pod template spec’s metadata. The deployment’s metadata is ignored. Label value of true forces the sidecar to be injected while a value of false forces the sidecar to not be injected.

    The following label overrides whatever the default policy was to force the sidecar to be injected:

    1. $ kubectl get deployment curl -o yaml | grep "sidecar.istio.io/inject:" -B4
    2. template:
    3.   metadata:
    4.     labels:
    5.       app: curl
    6.       sidecar.istio.io/inject: "true"

Pods cannot be created at all

Run kubectl describe -n namespace deployment name on the failing pod’s deployment. Failure to invoke the injection webhook will typically be captured in the event log.

  1. Warning  FailedCreate  3m (x17 over 8m)  replicaset-controller  Error creating: Internal error occurred: \
  2.     failed calling admission webhook "sidecar-injector.istio.io": Post https://istiod.istio-system.svc:443/inject: \
  3.     x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying \
  4.     to verify candidate authority certificate "Kubernetes.cluster.local")

x509: certificate signed by unknown authority errors are typically caused by an empty caBundle in the webhook configuration.

Verify the caBundle in the mutatingwebhookconfiguration matches the root certificate mounted in the istiod pod.

  1. $ kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | md5sum
  2. 4b95d2ba22ce8971c7c92084da31faf0  -
  3. $ kubectl -n istio-system get configmap istio-ca-root-cert -o jsonpath='{.data.root-cert\.pem}' | base64 -w 0 | md5sum
  4. 4b95d2ba22ce8971c7c92084da31faf0  -

The CA certificate should match. If they do not, restart the istiod pods.

  1. $ kubectl -n istio-system patch deployment istiod \
  2.     -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"date\":\"`date +'%s'`\"}}}}}"
  3. deployment.extensions "istiod" patched

Errors in deployment status

When automatic sidecar injection is enabled for a pod, and the injection fails for any reason, the pod creation will also fail. In such cases, you can check the deployment status of the pod to identify the error. The errors will also appear in the events of the namespace associated with the deployment.

For example, if the istiod control plane pod was not running when you tried to deploy your pod, the events would show the following error:

  1. $ kubectl get events -n curl
  2. ...
  3. 23m Normal   SuccessfulCreate replicaset/curl-9454cc476   Created pod: curl-9454cc476-khp45
  4. 22m Warning  FailedCreate     replicaset/curl-9454cc476   Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.96.44.51:443: connect: connection refused
  1. $ kubectl -n istio-system get pod -lapp=istiod
  2. NAME                            READY     STATUS    RESTARTS   AGE
  3. istiod-7d46d8d9db-jz2mh         1/1       Running     0         2d
  1. $ kubectl -n istio-system get endpoints istiod
  2. NAME           ENDPOINTS                                                  AGE
  3. istiod   10.244.2.8:15012,10.244.2.8:15010,10.244.2.8:15017 + 1 more...   3h18m

If the istiod pod or endpoints aren’t ready, check the pod logs and status for any indication about why the webhook pod is failing to start and serve traffic.

  1. $ for pod in $(kubectl -n istio-system get pod -lapp=istiod -o jsonpath='{.items[*].metadata.name}'); do \
  2.     kubectl -n istio-system logs ${pod} \
  3. done
  6. $ for pod in $(kubectl -n istio-system get pod -l app=istiod -o name); do \
  7. kubectl -n istio-system describe ${pod}; \
  8. done
  9. $

Automatic sidecar injection fails if the Kubernetes API server has proxy settings

When the Kubernetes API server includes proxy settings such as:

  1. env:
  2.   - name: http_proxy
  3.     value: http://proxy-wsa.esl.foo.com:80
  4.   - name: https_proxy
  5.     value: http://proxy-wsa.esl.foo.com:80
  6.   - name: no_proxy
  7.     value: 127.0.0.1,localhost,dockerhub.foo.com,devhub-docker.foo.com,10.84.100.125,10.84.100.126,10.84.100.127

With these settings, Sidecar injection fails. The only related failure log can be found in kube-apiserver log:

  1. W0227 21:51:03.156818       1 admission.go:257] Failed calling webhook, failing open sidecar-injector.istio.io: failed calling admission webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject: Service Unavailable

Make sure both pod and service CIDRs are not proxied according to *_proxy variables. Check the kube-apiserver files and logs to verify the configuration and whether any requests are being proxied.

One workaround is to remove the proxy settings from the kube-apiserver manifest, another workaround is to include istio-sidecar-injector.istio-system.svc or .svc in the no_proxy value. Make sure that kube-apiserver is restarted after each workaround.

An issue was filed with Kubernetes related to this and has since been closed. https://github.com/kubernetes/kubernetes/pull/58698#discussion_r163879443

Limitations for using Tcpdump in pods

Tcpdump doesn’t work in the sidecar pod - the container doesn’t run as root. However any other container in the same pod will see all the packets, since the network namespace is shared. iptables will also see the pod-wide configuration.

Communication between Envoy and the app happens on 127.0.0.1, and is not encrypted.

Cluster is not scaled down automatically

Due to the fact that the sidecar container mounts a local storage volume, the node autoscaler is unable to evict nodes with the injected pods. This is a known issue. The workaround is to add a pod annotation "cluster-autoscaler.kubernetes.io/safe-to-evict": "true" to the injected pods.

Pod or containers start with network issues if istio-proxy is not ready

Many applications execute commands or checks during startup, which require network connectivity. This can cause application containers to hang or restart if the istio-proxy sidecar container is not ready.

To avoid this, set holdApplicationUntilProxyStarts to true. This causes the sidecar injector to inject the sidecar at the start of the pod’s container list, and configures it to block the start of all other containers until the proxy is ready.

This can be added as a global config option:

  1. values.global.proxy.holdApplicationUntilProxyStarts: true

or as a pod annotation:

  1. proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'