Debugging Envoy and Pilot

Istio provides two very valuable commands to help diagnose traffic management configuration problems,the proxy-statusand proxy-config commands. The proxy-status commandallows you to get an overview of your mesh and identify the proxy causing the problem. Then proxy-config can be usedto inspect Envoy configuration and diagnose the issue.

If you want to try the commands described below, you can either:

  • Have a Kubernetes cluster with Istio and Bookinfo installed (e.g use istio.yaml as described ininstallation steps andBookinfo installation steps).OR

  • Use similar commands against your own application running in a Kubernetes cluster.

Get an overview of your mesh

The proxy-status command allows you to get an overview of your mesh. If you suspect one of your sidecars isn’treceiving configuration or is out of sync then proxy-status will tell you this.

  1. $ istioctl proxy-status
  2. PROXY                                                  CDS        LDS        EDS               RDS          PILOT                            VERSION
  3. details-v1-6dcc6fbb9d-wsjz4.default                    SYNCED     SYNCED     SYNCED     SYNCED       istio-pilot-75bdf98789-tfdvh     1.1.2
  4. istio-egressgateway-c49694485-l9d5l.istio-system       SYNCED     SYNCED     SYNCED     NOT SENT     istio-pilot-75bdf98789-tfdvh     1.1.2
  5. istio-ingress-6458b8c98f-7ks48.istio-system            SYNCED     SYNCED     SYNCED     NOT SENT     istio-pilot-75bdf98789-n2kqh     1.1.2
  6. istio-ingressgateway-7d6874b48f-qxhn5.istio-system     SYNCED     SYNCED     SYNCED     SYNCED       istio-pilot-75bdf98789-n2kqh     1.1.2
  7. productpage-v1-6c886ff494-hm7zk.default                SYNCED     SYNCED     SYNCED     STALE        istio-pilot-75bdf98789-n2kqh     1.1.2
  8. ratings-v1-5d9ff497bb-gslng.default                    SYNCED     SYNCED     SYNCED     SYNCED       istio-pilot-75bdf98789-n2kqh     1.1.2
  9. reviews-v1-55d4c455db-zjj2m.default                    SYNCED     SYNCED     SYNCED     SYNCED       istio-pilot-75bdf98789-n2kqh     1.1.2
  10. reviews-v2-686bbb668-99j76.default                     SYNCED     SYNCED     SYNCED     SYNCED       istio-pilot-75bdf98789-tfdvh     1.1.2
  11. reviews-v3-7b9b5fdfd6-4r52s.default                    SYNCED     SYNCED     SYNCED     SYNCED       istio-pilot-75bdf98789-n2kqh     1.1.2

If a proxy is missing from this list it means that it is not currently connected to a Pilot instance so will not bereceiving any configuration.

  • SYNCED means that Envoy has acknowledged the last configuration Pilot has sent to it.
  • NOT SENT means that Pilot hasn’t sent anything to Envoy. This usually is because Pilot has nothing to send.
  • STALE means that Pilot has sent an update to Envoy but has not received an acknowledgement. This usually indicatesa networking issue between Envoy and Pilot or a bug with Istio itself.

Retrieve diffs between Envoy and Istio Pilot

The proxy-status command can also be used to retrieve a diff between the configuration Envoy has loaded and theconfiguration Pilot would send, by providing a proxy ID. This can help you determine exactly what is out of sync andwhere the issue may lie.

  1. $ istioctl proxy-status details-v1-6dcc6fbb9d-wsjz4.default
  2. --- Pilot Clusters
  3. +++ Envoy Clusters
  4. @@ -374,36 +374,14 @@
  5.              "edsClusterConfig": {
  6.                 "edsConfig": {
  7.                    "ads": {
  9.                    }
  10.                 },
  11.                 "serviceName": "outbound|443||public-cr0bdc785ce3f14722918080a97e1f26be-alb1.kube-system.svc.cluster.local"
  12. -            },
  13. -            "connectTimeout": "1.000s",
  14. -            "circuitBreakers": {
  15. -               "thresholds": [
  16. -                  {
  17. -
  18. -                  }
  19. -               ]
  20. -            }
  21. -         }
  22. -      },
  23. -      {
  24. -         "cluster": {
  25. -            "name": "outbound|53||kube-dns.kube-system.svc.cluster.local",
  26. -            "type": "EDS",
  27. -            "edsClusterConfig": {
  28. -               "edsConfig": {
  29. -                  "ads": {
  30. -
  31. -                  }
  32. -               },
  33. -               "serviceName": "outbound|53||kube-dns.kube-system.svc.cluster.local"
  34.              },
  35.              "connectTimeout": "1.000s",
  36.              "circuitBreakers": {
  37.                 "thresholds": [
  38.                    {
  40.                    }
  42. Listeners Match
  43. Routes Match

Here you can see that the listeners and routes match but the clusters are out of sync.

Deep dive into Envoy configuration

The proxy-config command can be used to see how a given Envoy instance is configured. This can then be used topinpoint any issues you are unable to detect by just looking through your Istio configuration and custom resources.To get a basic summary of clusters, listeners or routes for a given pod use the command as follows (changing clustersfor listeners or routes when required):

  1. $ istioctl proxy-config cluster -n istio-system istio-ingressgateway-7d6874b48f-qxhn5
  2. SERVICE FQDN                                                                     PORT      SUBSET     DIRECTION     TYPE
  3. BlackHoleCluster                                                                 -         -          -             STATIC
  4. details.default.svc.cluster.local                                                9080      -          outbound      EDS
  5. heapster.kube-system.svc.cluster.local                                           80        -          outbound      EDS
  6. istio-citadel.istio-system.svc.cluster.local                                     8060      -          outbound      EDS
  7. istio-citadel.istio-system.svc.cluster.local                                     15014     -          outbound      EDS
  8. istio-egressgateway.istio-system.svc.cluster.local                               80        -          outbound      EDS
  9. ...

In order to debug Envoy you need to understand Envoy clusters/listeners/routes/endpoints and how they all interact.We will use the proxy-config command with the -o json and filtering flags to follow Envoy as it determines whereto send a request from the productpage pod to the reviews pod at reviews:9080.

  • If you query the listener summary on a pod you will notice Istio generates the following listeners:

    • A listener on 0.0.0.0:15001 that receives all traffic into and out of the pod, then hands the request over toa virtual listener.
    • A virtual listener per service IP, per each non-HTTP for outbound TCP/HTTPS traffic.
    • A virtual listener on the pod IP for each exposed port for inbound traffic.
    • A virtual listener on 0.0.0.0 per each HTTP port for outbound HTTP traffic.
  1. $ istioctl proxy-config listeners productpage-v1-6c886ff494-7vxhs
  2. ADDRESS            PORT      TYPE
  3. 172.21.252.250     15005     TCP <--+
  4. 172.21.252.250     15011     TCP    |
  5. 172.21.79.56       42422     TCP    |
  6. 172.21.160.5       443       TCP    |
  7. 172.21.157.6       443       TCP    |
  8. 172.21.117.222     443       TCP    |
  9. 172.21.0.10        53        TCP    |
  10. 172.21.126.131     443       TCP    |   Receives outbound non-HTTP traffic for relevant IP:PORT pair from listener `0.0.0.0_15001`
  11. 172.21.160.5       31400     TCP    |
  12. 172.21.81.159      9102      TCP    |
  13. 172.21.0.1         443       TCP    |
  14. 172.21.126.131     80        TCP    |
  15. 172.21.119.8       443       TCP    |
  16. 172.21.112.64      80        TCP    |
  17. 172.21.179.54      443       TCP    |
  18. 172.21.165.197     443       TCP <--+
  19. 0.0.0.0            9090      HTTP <-+
  20. 0.0.0.0            8060      HTTP   |
  21. 0.0.0.0            15010     HTTP   |
  22. 0.0.0.0            15003     HTTP   |
  23. 0.0.0.0            15004     HTTP   |
  24. 0.0.0.0            15014     HTTP   |   Receives outbound HTTP traffic for relevant port from listener `0.0.0.0_15001`
  25. 0.0.0.0            15007     HTTP   |
  26. 0.0.0.0            8080      HTTP   |
  27. 0.0.0.0            9091      HTTP   |
  28. 0.0.0.0            9080      HTTP   |
  29. 0.0.0.0            80        HTTP <-+
  30. 0.0.0.0            15001     TCP    // Receives all inbound and outbound traffic to the pod from IP tables and hands over to virtual listener
  31. 172.30.164.190     9080      HTTP   // Receives all inbound traffic on 9080 from listener `0.0.0.0_15001`
  • From the above summary you can see that every sidecar has a listener bound to 0.0.0.0:15001 which is whereIP tables routes all inbound and outbound pod traffic to. This listener has useOriginalDst set to true which meansit hands the request over to the listener that best matches the original destination of the request.If it can’t find any matching virtual listeners it sends the request to the PassthroughCluster which connects to the destination directly.
  1. $ istioctl proxy-config listeners productpage-v1-6c886ff494-7vxhs --port 15001 -o json
  2. [
  3.     {
  4.         "name": "virtual",
  5.         "address": {
  6.             "socketAddress": {
  7.                 "address": "0.0.0.0",
  8.                 "portValue": 15001
  9.             }
  10.         },
  11.         "filterChains": [
  12.             {
  13.                 "filters": [
  14.                     {
  15.                         "name": "envoy.tcp_proxy",
  16.                         "config": {
  17.                             "cluster": "PassthroughCluster",
  18.                             "stat_prefix": "PassthroughCluster"
  19.                         }
  20.                     }
  21.                 ]
  22.             }
  23.         ],
  24.         "useOriginalDst": true
  25.     }
  26. ]
  • Our request is an outbound HTTP request to port 9080 this means it gets handed off to the 0.0.0.0:9080 virtuallistener. This listener then looks up the route configuration in its configured RDS. In this case it will be lookingup route 9080 in RDS configured by Pilot (via ADS).
  1. $ istioctl proxy-config listeners productpage-v1-6c886ff494-7vxhs -o json --address 0.0.0.0 --port 9080
  2. ...
  3. "rds": {
  4.     "config_source": {
  5.         "ads": {}
  6.     },
  7.     "route_config_name": "9080"
  8. }
  9. ...
  • The 9080 route configuration only has a virtual host for each service. Our request is heading to the reviewsservice so Envoy will select the virtual host to which our request matches a domain. Once matched on domain Envoylooks for the first route that matches the request. In this case we don’t have any advanced routing so there is onlyone route that matches on everything. This route tells Envoy to send the request to theoutbound|9080||reviews.default.svc.cluster.local cluster.
  1. $ istioctl proxy-config routes productpage-v1-6c886ff494-7vxhs --name 9080 -o json
  2. [
  3.     {
  4.         "name": "9080",
  5.         "virtualHosts": [
  6.             {
  7.                 "name": "reviews.default.svc.cluster.local:9080",
  8.                 "domains": [
  9.                     "reviews.default.svc.cluster.local",
  10.                     "reviews.default.svc.cluster.local:9080",
  11.                     "reviews",
  12.                     "reviews:9080",
  13.                     "reviews.default.svc.cluster",
  14.                     "reviews.default.svc.cluster:9080",
  15.                     "reviews.default.svc",
  16.                     "reviews.default.svc:9080",
  17.                     "reviews.default",
  18.                     "reviews.default:9080",
  19.                     "172.21.152.34",
  20.                     "172.21.152.34:9080"
  21.                 ],
  22.                 "routes": [
  23.                     {
  24.                         "match": {
  25.                             "prefix": "/"
  26.                         },
  27.                         "route": {
  28.                             "cluster": "outbound|9080||reviews.default.svc.cluster.local",
  29.                             "timeout": "0.000s"
  30.                         },
  31. ...
  • This cluster is configured to retrieve the associated endpoints from Pilot (via ADS). So Envoy will then use theserviceName field as a key to look up the list of Endpoints and proxy the request to one of them.
  1. $ istioctl proxy-config cluster productpage-v1-6c886ff494-7vxhs --fqdn reviews.default.svc.cluster.local -o json
  2. [
  3.     {
  4.         "name": "outbound|9080||reviews.default.svc.cluster.local",
  5.         "type": "EDS",
  6.         "edsClusterConfig": {
  7.             "edsConfig": {
  8.                 "ads": {}
  9.             },
  10.             "serviceName": "outbound|9080||reviews.default.svc.cluster.local"
  11.         },
  12.         "connectTimeout": "1.000s",
  13.         "circuitBreakers": {
  14.             "thresholds": [
  15.                 {}
  16.             ]
  17.         }
  18.     }
  19. ]
  • To see the endpoints currently available for this cluster use the proxy-config endpoints command.
  1. $ istioctl proxy-config endpoints productpage-v1-6c886ff494-7vxhs --cluster "outbound|9080||reviews.default.svc.cluster.local"
  2. ENDPOINT             STATUS      OUTLIER CHECK     CLUSTER
  3. 172.17.0.17:9080     HEALTHY     OK                outbound|9080||reviews.default.svc.cluster.local
  4. 172.17.0.18:9080     HEALTHY     OK                outbound|9080||reviews.default.svc.cluster.local
  5. 172.17.0.5:9080      HEALTHY     OK                outbound|9080||reviews.default.svc.cluster.local

Inspecting Bootstrap configuration

So far we have looked at configuration retrieved (mostly) from Pilot, however Envoy requires some bootstrap configuration thatincludes information like where Pilot can be found. To view this use the following command:

  1. $ istioctl proxy-config bootstrap -n istio-system istio-ingressgateway-7d6874b48f-qxhn5
  2. {
  3.     "bootstrap": {
  4.         "node": {
  5.             "id": "router~172.30.86.14~istio-ingressgateway-7d6874b48f-qxhn5.istio-system~istio-system.svc.cluster.local",
  6.             "cluster": "istio-ingressgateway",
  7.             "metadata": {
  8.                     "POD_NAME": "istio-ingressgateway-7d6874b48f-qxhn5",
  9.                     "istio": "sidecar"
  10.                 },
  11.             "buildVersion": "0/1.8.0-dev//RELEASE"
  12.         },
  13. ...

Verifying connectivity to Istio Pilot

Verifying connectivity to Pilot is a useful troubleshooting step. Every proxy container in the service mesh should be able to communicate with Pilot. This can be accomplished in a few simple steps:

  • Get the name of the Istio Ingress pod:
  1. $ INGRESS_POD_NAME=$(kubectl get po -n istio-system | grep ingressgateway\- | awk '{print$1}'); echo ${INGRESS_POD_NAME};
  • Exec into the Istio Ingress pod:
  1. $ kubectl exec -it $INGRESS_POD_NAME -n istio-system /bin/bash
  • Test connectivity to Pilot using curl. The following example invokes the v1 registration API using default Pilot configuration parameters and mutual TLS enabled:
  1. $ curl -k --cert /etc/certs/cert-chain.pem --cacert /etc/certs/root-cert.pem --key /etc/certs/key.pem https://istio-pilot:8080/debug/edsz

If mutual TLS is disabled:

  1. $ curl http://istio-pilot:8080/debug/edsz

You should receive a response listing the “service-key” and “hosts” for each service in the mesh.

What Envoy version is Istio using?

To find out the Envoy version used in deployment, you can exec into the container and query the server_info endpoint:

  1. $ kubectl exec -it PODNAME -c istio-proxy -n NAMESPACE pilot-agent request GET server_info
  2. {
  3.  "version": "48bc83d8f0582fc060ef76d5aa3d75400e739d9e/1.12.0-dev/Clean/RELEASE/BoringSSL"
  4. }

相关内容

Mixer Configuration Model

Describes the configuration model for Istio's policy enforcement and telemetry mechanisms.

Traffic Management

Describes the various Istio features focused on traffic routing and control.

可观察性

描述 Istio 提供的遥测和监控特性。

安装 Istio CNI 插件

安装并使用 Istio CNI 插件，可以让运维人员用更低的权限来部署服务。