Troubleshoot issues with waypoints

This guide describes what to do if you have enrolled a namespace, service or workload in a waypoint proxy, but you are not seeing the expected behavior.

Problems with traffic routing or security policy

To send some requests to the reviews service via the productpage service from the curl pod:

  1. $ kubectl exec deploy/curl -- curl -s http://productpage:9080/productpage

To send some requests to the reviews v2 pod from the curl pod:

  1. $ export REVIEWS_V2_POD_IP=$(kubectl get pod -l version=v2,app=reviews -o jsonpath='{.items[0].status.podIP}')
  2. $ kubectl exec deploy/curl -- curl -s http://$REVIEWS_V2_POD_IP:9080/reviews/1

Requests to the reviews service should be enforced by the reviews-svc-waypoint for any L7 policies. Requests to the reviews v2 pod should be enforced by the reviews-v2-pod-waypoint for any L7 policies.

  1. If your L7 configuration isn’t applied, run istioctl analyze first to check if your configuration has a validation issue.

    1. $ istioctl analyze
    2. ✔ No validation issues found when analyzing namespace: default.

  2. Determine which waypoint is implementing the L7 configuration for your service or pod.

    If your source calls the destination using the service’s hostname or IP, use the istioctl experimental ztunnel-config service command to confirm your waypoint is used by the destination service. Following the example earlier, the reviews service should use the reviews-svc-waypoint while all other services in the default namespace should use the namespace waypoint.

    1. $ istioctl ztunnel-config service
    2. NAMESPACE    SERVICE NAME            SERVICE VIP   WAYPOINT
    3. default      bookinfo-gateway-istio  10.43.164.194 waypoint
    4. default      bookinfo-gateway-istio  10.43.164.194 waypoint
    5. default      bookinfo-gateway-istio  10.43.164.194 waypoint
    6. default      bookinfo-gateway-istio  10.43.164.194 waypoint
    7. default      details                 10.43.160.119 waypoint
    8. default      kubernetes              10.43.0.1     waypoint
    9. default      productpage             10.43.172.254 waypoint
    10. default      ratings                 10.43.71.236  waypoint
    11. default      reviews                 10.43.162.105 reviews-svc-waypoint
    12. ...

    If your source calls the destination using a pod IP, use the istioctl ztunnel-config workload command to confirm your waypoint is used by the destination pod. Following the example earlier, the reviews v2 pod should use the reviews-v2-pod-waypoint while all other pods in the default namespace should not have any waypoints, because by default a waypoint only handles traffic addressed to services.

    1. $ istioctl ztunnel-config workload
    2. NAMESPACE    POD NAME                                    IP         NODE                     WAYPOINT                PROTOCOL
    3. default      bookinfo-gateway-istio-7c57fc4647-wjqvm     10.42.2.8  k3d-k3s-default-server-0 None                    TCP
    4. default      details-v1-698d88b-wwsnv                    10.42.2.4  k3d-k3s-default-server-0 None                    HBONE
    5. default      productpage-v1-675fc69cf-fp65z              10.42.2.6  k3d-k3s-default-server-0 None                    HBONE
    6. default      ratings-v1-6484c4d9bb-crjtt                 10.42.0.4  k3d-k3s-default-agent-0  None                    HBONE
    7. default      reviews-svc-waypoint-c49f9f569-b492t        10.42.2.10 k3d-k3s-default-server-0 None                    TCP
    8. default      reviews-v1-5b5d6494f4-nrvfx                 10.42.2.5  k3d-k3s-default-server-0 None                    HBONE
    9. default      reviews-v2-5b667bcbf8-gj7nz                 10.42.0.5  k3d-k3s-default-agent-0  reviews-v2-pod-waypoint HBONE
    10. ...

    If the value for the pod’s waypoint column isn’t correct, verify your pod is labeled with istio.io/use-waypoint and the label’s value is the name of a waypoint that can process workload traffic. For example, if your reviews v2 pod uses a waypoint that can only process service traffic, you will not see any waypoint used by that pod. If the istio.io/use-waypoint label on your pod looks correct verify that the Gateway resource for your waypoint is labeled with a compatible value for istio.io/waypoint-for. In the case of a pod, suitable values would be all or workload.

  3. Check the waypoint’s proxy status via the istioctl proxy-status command.

    1. $ istioctl proxy-status
    2. NAME                                                CLUSTER        CDS         LDS         EDS          RDS          ECDS         ISTIOD                      VERSION
    3. bookinfo-gateway-istio-7c57fc4647-wjqvm.default     Kubernetes     SYNCED      SYNCED      SYNCED       SYNCED       NOT SENT     istiod-795d55fc6d-vqtjx     1.23-alpha.75c6eafc5bc8d160b5643c3ea18acb9785855564
    4. reviews-svc-waypoint-c49f9f569-b492t.default        Kubernetes     SYNCED      SYNCED      SYNCED       NOT SENT     NOT SENT     istiod-795d55fc6d-vqtjx     1.23-alpha.75c6eafc5bc8d160b5643c3ea18acb9785855564
    5. reviews-v2-pod-waypoint-7f5dbd597-7zzw7.default     Kubernetes     SYNCED      SYNCED      NOT SENT     NOT SENT     NOT SENT     istiod-795d55fc6d-vqtjx     1.23-alpha.75c6eafc5bc8d160b5643c3ea18acb9785855564
    6. waypoint-6f7b665c89-6hppr.default                   Kubernetes     SYNCED      SYNCED      SYNCED       NOT SENT     NOT SENT     istiod-795d55fc6d-vqtjx     1.23-alpha.75c6eafc5bc8d160b5643c3ea18acb9785855564
    7. ...

  4. Enable Envoy’s access log and check the logs of the waypoint proxy after sending some requests:

    1. $ kubectl logs deploy/waypoint

    If there is not enough information, you can enable the debug logs for the waypoint proxy:

    1. $ istioctl pc log deploy/waypoint --level debug

  5. Check the envoy configuration for the waypoint via the istioctl proxy-config command, which shows all the information related to the waypoint such as clusters, endpoints, listeners, routes and secrets:

    1. $ istioctl proxy-config all deploy/waypoint

Refer to the deep dive into Envoy configuration section for more information regarding how to debug Envoy since waypoint proxies are based on Envoy.