Remotely Accessing Telemetry Addons

This task shows how to configure Istio to expose and access the telemetry addons outside ofa cluster.

Configuring remote access

Remote access to the telemetry addons can be configured in a number of different ways. This task coverstwo basic access methods: secure (via HTTPS) and insecure (via HTTP). The secure method is stronglyrecommended for any production or sensitive environment. Insecure access is simpler to set up, butwill not protect any credentials or data transmitted outside of your cluster.

Option 1: Secure access (HTTPS)

A server certificate is required for secure access. Follow these steps to install and configureserver certificates for a domain that you control.

You may use self-signed certificates instead. Visit ourSecuring Gateways with HTTPS Using Secret Discovery Service taskfor general information on using self-signed certificates to access in-cluster services.

This option covers securing the transport layer only. You should also configure the telemetryaddons to require authentication when exposing them externally.

To install Istio accordingly, use the following installation options:

  • —set values.gateways.enabled=true
  • —set values.gateways.istio-ingressgateway.enabled=true

  • —set values.gateways.istio-ingressgateway.sds.enabled=trueTo additionally install the telemetry addons, use the following installation options:

  • Grafana: —set values.grafana.enabled=true

  • Kiali: —set values.kiali.enabled=true
  • Prometheus: —set values.prometheus.enabled=true

  • Tracing: —set values.tracing.enabled=true

    • Configure the DNS records for your domain.

  • Get the external IP address of the istio-ingressgateway.

  1. $ kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
  2. <IP ADDRESS OF CLUSTER INGRESS>
  • Set an environment variable to hold your target domain.
  1. $ TELEMETRY_DOMAIN=<your.desired.domain>
  • Point your desired domain at that external IP address via your domain provider.

The mechanism for achieving this step varies by provider. Here are a few example documentation links:

  1. - Bluehost: [DNS Management Add Edit or Delete DNS Entries](https://my.bluehost.com/hosting/help/559)
  2. - GoDaddy: [Add an A record](https://www.godaddy.com/help/add-an-a-record-19238)
  3. - Google Domains: [Resource Records](https://support.google.com/domains/answer/3290350?hl=en)
  4. - Name.com: [Adding an A record](https://www.name.com/support/articles/115004893508-Adding-an-A-record)
  • Verify that the DNS records are correct.
  1. $ dig +short $TELEMETRY_DOMAIN
  2. <IP ADDRESS OF CLUSTER INGRESS>
  • Generate a server certificate
  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: certmanager.k8s.io/v1alpha1
  3. kind: Certificate
  4. metadata:
  5.   name: telemetry-gw-cert
  6.   namespace: istio-system
  7. spec:
  8.   secretName: telemetry-gw-cert
  9.   issuerRef:
  10.     name: letsencrypt
  11.     kind: ClusterIssuer
  12.   commonName: $TELEMETRY_DOMAIN
  13.   dnsNames:
  14.   - $TELEMETRY_DOMAIN
  15.   acme:
  16.     config:
  17.     - http01:
  18.         ingressClass: istio
  19.       domains:
  20.       - $TELEMETRY_DOMAIN
  21. ---
  22. EOF
  23. certificate.certmanager.k8s.io "telemetry-gw-cert" created
  • Wait until the server certificate is ready.
  1. $ JSONPATH='{range .items[*]}{@.metadata.name}:{range @.status.conditions[*]}{@.type}={@.status}{end}{end}' && kubectl -n istio-system get certificates -o jsonpath="$JSONPATH"
  2. telemetry-gw-cert:Ready=True

  • Apply networking configuration for the telemetry addons.

    • Apply the following configuration to expose Grafana:
  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: Gateway
  4. metadata:
  5.   name: grafana-gateway
  6.   namespace: istio-system
  7. spec:
  8.   selector:
  9.     istio: ingressgateway
  10.   servers:
  11.   - port:
  12.       number: 15031
  13.       name: https-grafana
  14.       protocol: HTTPS
  15.     tls:
  16.       mode: SIMPLE
  17.       serverCertificate: sds
  18.       privateKey: sds
  19.       credentialName: telemetry-gw-cert
  20.     hosts:
  21.     - "$TELEMETRY_DOMAIN"
  22. ---
  23. apiVersion: networking.istio.io/v1alpha3
  24. kind: VirtualService
  25. metadata:
  26.   name: grafana-vs
  27.   namespace: istio-system
  28. spec:
  29.   hosts:
  30.   - "$TELEMETRY_DOMAIN"
  31.   gateways:
  32.   - grafana-gateway
  33.   http:
  34.   - match:
  35.     - port: 15031
  36.     route:
  37.     - destination:
  38.         host: grafana
  39.         port:
  40.           number: 3000
  41. ---
  42. apiVersion: networking.istio.io/v1alpha3
  43. kind: DestinationRule
  44. metadata:
  45.   name: grafana
  46.   namespace: istio-system
  47. spec:
  48.   host: grafana
  49.   trafficPolicy:
  50.     tls:
  51.       mode: DISABLE
  52. ---
  53. EOF
  54. gateway.networking.istio.io "grafana-gateway" configured
  55. virtualservice.networking.istio.io "grafana-vs" configured
  56. destinationrule.networking.istio.io "grafana" configured
  • Apply the following configuration to expose Kiali:
  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: Gateway
  4. metadata:
  5.   name: kiali-gateway
  6.   namespace: istio-system
  7. spec:
  8.   selector:
  9.     istio: ingressgateway
  10.   servers:
  11.   - port:
  12.       number: 15029
  13.       name: https-kiali
  14.       protocol: HTTPS
  15.     tls:
  16.       mode: SIMPLE
  17.       serverCertificate: sds
  18.       privateKey: sds
  19.       credentialName: telemetry-gw-cert
  20.     hosts:
  21.     - "$TELEMETRY_DOMAIN"
  22. ---
  23. apiVersion: networking.istio.io/v1alpha3
  24. kind: VirtualService
  25. metadata:
  26.   name: kiali-vs
  27.   namespace: istio-system
  28. spec:
  29.   hosts:
  30.   - "$TELEMETRY_DOMAIN"
  31.   gateways:
  32.   - kiali-gateway
  33.   http:
  34.   - match:
  35.     - port: 15029
  36.     route:
  37.     - destination:
  38.         host: kiali
  39.         port:
  40.           number: 20001
  41. ---
  42. apiVersion: networking.istio.io/v1alpha3
  43. kind: DestinationRule
  44. metadata:
  45.   name: kiali
  46.   namespace: istio-system
  47. spec:
  48.   host: kiali
  49.   trafficPolicy:
  50.     tls:
  51.       mode: DISABLE
  52. ---
  53. EOF
  54. gateway.networking.istio.io "kiali-gateway" configured
  55. virtualservice.networking.istio.io "kiali-vs" configured
  56. destinationrule.networking.istio.io "kiali" configured
  • Apply the following configuration to expose Prometheus:
  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: Gateway
  4. metadata:
  5.   name: prometheus-gateway
  6.   namespace: istio-system
  7. spec:
  8.   selector:
  9.     istio: ingressgateway
  10.   servers:
  11.   - port:
  12.       number: 15030
  13.       name: https-prom
  14.       protocol: HTTPS
  15.     tls:
  16.       mode: SIMPLE
  17.       serverCertificate: sds
  18.       privateKey: sds
  19.       credentialName: telemetry-gw-cert
  20.     hosts:
  21.     - "$TELEMETRY_DOMAIN"
  22. ---
  23. apiVersion: networking.istio.io/v1alpha3
  24. kind: VirtualService
  25. metadata:
  26.   name: prometheus-vs
  27.   namespace: istio-system
  28. spec:
  29.   hosts:
  30.   - "$TELEMETRY_DOMAIN"
  31.   gateways:
  32.   - prometheus-gateway
  33.   http:
  34.   - match:
  35.     - port: 15030
  36.     route:
  37.     - destination:
  38.         host: prometheus
  39.         port:
  40.           number: 9090
  41. ---
  42. apiVersion: networking.istio.io/v1alpha3
  43. kind: DestinationRule
  44. metadata:
  45.   name: prometheus
  46.   namespace: istio-system
  47. spec:
  48.   host: prometheus
  49.   trafficPolicy:
  50.     tls:
  51.       mode: DISABLE
  52. ---
  53. EOF
  54. gateway.networking.istio.io "prometheus-gateway" configured
  55. virtualservice.networking.istio.io "prometheus-vs" configured
  56. destinationrule.networking.istio.io "prometheus" configured
  • Apply the following configuration to expose the tracing service:
  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: Gateway
  4. metadata:
  5.   name: tracing-gateway
  6.   namespace: istio-system
  7. spec:
  8.   selector:
  9.     istio: ingressgateway
  10.   servers:
  11.   - port:
  12.       number: 15032
  13.       name: https-tracing
  14.       protocol: HTTPS
  15.     tls:
  16.       mode: SIMPLE
  17.       serverCertificate: sds
  18.       privateKey: sds
  19.       credentialName: telemetry-gw-cert
  20.     hosts:
  21.     - "$TELEMETRY_DOMAIN"
  22. ---
  23. apiVersion: networking.istio.io/v1alpha3
  24. kind: VirtualService
  25. metadata:
  26.   name: tracing-vs
  27.   namespace: istio-system
  28. spec:
  29.   hosts:
  30.   - "$TELEMETRY_DOMAIN"
  31.   gateways:
  32.   - tracing-gateway
  33.   http:
  34.   - match:
  35.     - port: 15032
  36.     route:
  37.     - destination:
  38.         host: tracing
  39.         port:
  40.           number: 80
  41. ---
  42. apiVersion: networking.istio.io/v1alpha3
  43. kind: DestinationRule
  44. metadata:
  45.   name: tracing
  46.   namespace: istio-system
  47. spec:
  48.   host: tracing
  49.   trafficPolicy:
  50.     tls:
  51.       mode: DISABLE
  52. ---
  53. EOF
  54. gateway.networking.istio.io "tracing-gateway" configured
  55. virtualservice.networking.istio.io "tracing-vs" configured
  56. destinationrule.networking.istio.io "tracing" configured

Option 2: Insecure access (HTTP)

  • Install Istio in your cluster with your desired telemetry addons.

To additionally install the telemetry addons, use the following installation options:

  • Grafana: —set values.grafana.enabled=true
  • Kiali: —set values.kiali.enabled=true
  • Prometheus: —set values.prometheus.enabled=true

  • Tracing: —set values.tracing.enabled=true

    • Apply networking configuration for the telemetry addons.

  • Apply the following configuration to expose Grafana:

  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: Gateway
  4. metadata:
  5.   name: grafana-gateway
  6.   namespace: istio-system
  7. spec:
  8.   selector:
  9.     istio: ingressgateway
  10.   servers:
  11.   - port:
  12.       number: 15031
  13.       name: http-grafana
  14.       protocol: HTTP
  15.     hosts:
  16.     - "*"
  17. ---
  18. apiVersion: networking.istio.io/v1alpha3
  19. kind: VirtualService
  20. metadata:
  21.   name: grafana-vs
  22.   namespace: istio-system
  23. spec:
  24.   hosts:
  25.   - "*"
  26.   gateways:
  27.   - grafana-gateway
  28.   http:
  29.   - match:
  30.     - port: 15031
  31.     route:
  32.     - destination:
  33.         host: grafana
  34.         port:
  35.           number: 3000
  36. ---
  37. apiVersion: networking.istio.io/v1alpha3
  38. kind: DestinationRule
  39. metadata:
  40.   name: grafana
  41.   namespace: istio-system
  42. spec:
  43.   host: grafana
  44.   trafficPolicy:
  45.     tls:
  46.       mode: DISABLE
  47. ---
  48. EOF
  49. gateway.networking.istio.io "grafana-gateway" configured
  50. virtualservice.networking.istio.io "grafana-vs" configured
  51. destinationrule.networking.istio.io "grafana" configured
  • Apply the following configuration to expose Kiali:
  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: Gateway
  4. metadata:
  5.   name: kiali-gateway
  6.   namespace: istio-system
  7. spec:
  8.   selector:
  9.     istio: ingressgateway
  10.   servers:
  11.   - port:
  12.       number: 15029
  13.       name: http-kiali
  14.       protocol: HTTP
  15.     hosts:
  16.     - "*"
  17. ---
  18. apiVersion: networking.istio.io/v1alpha3
  19. kind: VirtualService
  20. metadata:
  21.   name: kiali-vs
  22.   namespace: istio-system
  23. spec:
  24.   hosts:
  25.   - "*"
  26.   gateways:
  27.   - kiali-gateway
  28.   http:
  29.   - match:
  30.     - port: 15029
  31.     route:
  32.     - destination:
  33.         host: kiali
  34.         port:
  35.           number: 20001
  36. ---
  37. apiVersion: networking.istio.io/v1alpha3
  38. kind: DestinationRule
  39. metadata:
  40.   name: kiali
  41.   namespace: istio-system
  42. spec:
  43.   host: kiali
  44.   trafficPolicy:
  45.     tls:
  46.       mode: DISABLE
  47. ---
  48. EOF
  49. gateway.networking.istio.io "kiali-gateway" configured
  50. virtualservice.networking.istio.io "kiali-vs" configured
  51. destinationrule.networking.istio.io "kiali" configured
  • Apply the following configuration to expose Prometheus:
  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: Gateway
  4. metadata:
  5.   name: prometheus-gateway
  6.   namespace: istio-system
  7. spec:
  8.   selector:
  9.     istio: ingressgateway
  10.   servers:
  11.   - port:
  12.       number: 15030
  13.       name: http-prom
  14.       protocol: HTTP
  15.     hosts:
  16.     - "*"
  17. ---
  18. apiVersion: networking.istio.io/v1alpha3
  19. kind: VirtualService
  20. metadata:
  21.   name: prometheus-vs
  22.   namespace: istio-system
  23. spec:
  24.   hosts:
  25.   - "*"
  26.   gateways:
  27.   - prometheus-gateway
  28.   http:
  29.   - match:
  30.     - port: 15030
  31.     route:
  32.     - destination:
  33.         host: prometheus
  34.         port:
  35.           number: 9090
  36. ---
  37. apiVersion: networking.istio.io/v1alpha3
  38. kind: DestinationRule
  39. metadata:
  40.   name: prometheus
  41.   namespace: istio-system
  42. spec:
  43.   host: prometheus
  44.   trafficPolicy:
  45.     tls:
  46.       mode: DISABLE
  47. ---
  48. EOF
  49. gateway.networking.istio.io "prometheus-gateway" configured
  50. virtualservice.networking.istio.io "prometheus-vs" configured
  51. destinationrule.networking.istio.io "prometheus" configured
  • Apply the following configuration to expose the tracing service:
  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: networking.istio.io/v1alpha3
  3. kind: Gateway
  4. metadata:
  5.   name: tracing-gateway
  6.   namespace: istio-system
  7. spec:
  8.   selector:
  9.     istio: ingressgateway
  10.   servers:
  11.   - port:
  12.       number: 15032
  13.       name: http-tracing
  14.       protocol: HTTP
  15.     hosts:
  16.     - "*"
  17. ---
  18. apiVersion: networking.istio.io/v1alpha3
  19. kind: VirtualService
  20. metadata:
  21.   name: tracing-vs
  22.   namespace: istio-system
  23. spec:
  24.   hosts:
  25.   - "*"
  26.   gateways:
  27.   - tracing-gateway
  28.   http:
  29.   - match:
  30.     - port: 15032
  31.     route:
  32.     - destination:
  33.         host: tracing
  34.         port:
  35.           number: 80
  36. ---
  37. apiVersion: networking.istio.io/v1alpha3
  38. kind: DestinationRule
  39. metadata:
  40.   name: tracing
  41.   namespace: istio-system
  42. spec:
  43.   host: tracing
  44.   trafficPolicy:
  45.     tls:
  46.       mode: DISABLE
  47. ---
  48. EOF
  49. gateway.networking.istio.io "tracing-gateway" configured
  50. virtualservice.networking.istio.io "tracing-vs" configured
  51. destinationrule.networking.istio.io "tracing" configured

  • Visit the telemetry addons via your browser.

    • Kiali: http://<IP ADDRESS OF CLUSTER INGRESS>:15029/
    • Prometheus: http://<IP ADDRESS OF CLUSTER INGRESS>:15030/
    • Grafana: http://<IP ADDRESS OF CLUSTER INGRESS>:15031/
    • Tracing: http://<IP ADDRESS OF CLUSTER INGRESS>:15032/

Cleanup

  • Remove all related Gateways:
  1. $ kubectl -n istio-system delete gateway grafana-gateway kiali-gateway prometheus-gateway tracing-gateway
  2. gateway.networking.istio.io "grafana-gateway" deleted
  3. gateway.networking.istio.io "kiali-gateway" deleted
  4. gateway.networking.istio.io "prometheus-gateway" deleted
  5. gateway.networking.istio.io "tracing-gateway" deleted
  • Remove all related Virtual Services:
  1. $ kubectl -n istio-system delete virtualservice grafana-vs kiali-vs prometheus-vs tracing-vs
  2. virtualservice.networking.istio.io "grafana-vs" deleted
  3. virtualservice.networking.istio.io "kiali-vs" deleted
  4. virtualservice.networking.istio.io "prometheus-vs" deleted
  5. virtualservice.networking.istio.io "tracing-vs" deleted
  • If installed, remove the gateway certificate:
  1. $ kubectl -n istio-system delete certificate telemetry-gw-cert
  2. certificate.certmanager.k8s.io "telemetry-gw-cert" deleted

See also

Jaeger

Learn how to configure the proxies to send tracing requests to Jaeger.

Zipkin

Learn how to configure the proxies to send tracing requests to Zipkin.

LightStep

How to configure the proxies to send tracing requests to LightStep.

Overview

Overview of distributed tracing in Istio.

Multi-Mesh Deployments for Isolation and Boundary Protection

Deploy environments that require isolation into separate meshes and enable inter-mesh communication by mesh federation.

Secure Control of Egress Traffic in Istio, part 3

Comparison of alternative solutions to control egress traffic including performance considerations.