Debugging Virtual Machines

This page describes how to troubleshoot issues with Istio deployed to Virtual Machines. Before reading this, you should take the steps in Virtual Machine Installation. Additionally, Virtual Machine Architecture can help you understand how the components interact.

Troubleshooting an Istio Virtual Machine installation is similar to troubleshooting issues with proxies running inside Kubernetes, but there are some key differences to be aware of.

While much of the same information is available on both platforms, accessing this information differs.

Monitoring health

The Istio sidecar is typically run as a systemd unit. To ensure its running properly, you can check that status:

  1. $ systemctl status istio

Additionally, the sidecar health can be programmatically check at its health endpoint:

  1. $ curl localhost:15021/healthz/ready -I

Logs

Logs for the Istio proxy can be found in a few places.

To access the systemd logs, which has details about the initialization of the proxy:

  1. $ journalctl -f -u istio -n 1000

The proxy will redirect stderr and stdout to /var/log/istio/istio.err.log and /var/log/istio/istio.log, respectively. To view these in a format similar to kubectl:

  1. $ tail /var/log/istio/istio.err.log /var/log/istio/istio.log -Fq -n 100

Log levels can be modified by changing the cluster.env configuration file. Make sure to restart istio if it is already running:

  1. $ echo "ISTIO_AGENT_FLAGS=\"--log_output_level=dns:debug --proxyLogLevel=debug\"" >> /var/lib/istio/envoy/cluster.env
  2. $ systemctl restart istio

Iptables

To ensure iptables rules have been successfully applied:

  1. $ sudo iptables-save
  2. ...
  3. -A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
  4. -A ISTIO_OUTPUT -j ISTIO_REDIRECT

Istioctl

Most istioctl commands will function properly with virtual machines. For example, istioctl proxy-status can be used to view all connected proxies:

  1. $ istioctl proxy-status
  2. NAME           CDS        LDS        EDS        RDS      ISTIOD                    VERSION
  3. vm-1.default   SYNCED     SYNCED     SYNCED     SYNCED   istiod-789ffff8-f2fkt     1.20.0

However, istioctl proxy-config relies on functionality in Kubernetes to connect to a proxy, which will not work for virtual machines. Instead, a file containing the configuration dump from Envoy can be passed. For example:

  1. $ curl -s localhost:15000/config_dump | istioctl proxy-config clusters --file -
  2. SERVICE FQDN                            PORT      SUBSET  DIRECTION     TYPE
  3. istiod.istio-system.svc.cluster.local   443       -       outbound      EDS
  4. istiod.istio-system.svc.cluster.local   15010     -       outbound      EDS
  5. istiod.istio-system.svc.cluster.local   15012     -       outbound      EDS
  6. istiod.istio-system.svc.cluster.local   15014     -       outbound      EDS

Automatic registration

When a virtual machine connects to Istiod, a WorkloadEntry will automatically be created. This enables the virtual machine to become a part of a Service, similar to an Endpoint in Kubernetes.

To check these are created correctly:

  1. $ kubectl get workloadentries
  2. NAME             AGE   ADDRESS
  3. vm-10.128.0.50   14m   10.128.0.50

Certificates

Virtual machines handle certificates differently than Kubernetes Pods, which use a Kubernetes-provided service account token to authenticate and renew mTLS certificates. Instead, existing mTLS credentials are used to authenticate with the certificate authority and renew certificates.

The status of these certificates can be viewed in the same way as in Kubernetes:

  1. $ curl -s localhost:15000/config_dump | ./istioctl proxy-config secret --file -
  2. RESOURCE NAME     TYPE           STATUS     VALID CERT     SERIAL NUMBER                               NOT AFTER                NOT BEFORE
  3. default           Cert Chain     ACTIVE     true           251932493344649542420616421203546836446     2021-01-29T18:07:21Z     2021-01-28T18:07:21Z
  4. ROOTCA            CA             ACTIVE     true           81663936513052336343895977765039160718      2031-01-26T17:54:44Z     2021-01-28T17:54:44Z

Additionally, these are persisted to disk to ensure downtime or restarts do not lose state.

  1. $ ls /etc/certs
  2. cert-chain.pem  key.pem  root-cert.pem