Install Istio with an External Control Plane

Introduction

This guide walks you through the installation of an external control plane. The external control plane deployment model enables mesh operators to install and manage mesh control planes on separate external clusters. This deployment model allows a clear separation between mesh operators and mesh admins. The mesh operators can install and manage the Istio control planes while the mesh admins only need to configure the mesh resources.

This feature is currently considered alpha.

External control plane cluster and remote cluster

External control plane cluster and remote cluster

Requirements

Clusters

This guide requires that you have two Kubernetes clusters with any of the supported Kubernetes versions: 1.16, 1.17, 1.18, 1.19.

The first cluster contains the external control plane installed in the external-istiod namespace. An ingress gateway is also installed in the istio-system namespace to provide mesh sidecars access to the external control plane.

The second cluster is a remote cluster running the mesh workloads. Its Kubernetes API server also provides the configuration for the control plane (istiod) running in the external cluster.

API Server Access

The Kubernetes API server in the remote cluster must be accessible to the external control plane cluster. Many cloud providers make API servers publicly accessible via network load balancers (NLBs). If the API server is not directly accessible, you will have to modify the installation procedure to enable access. For example, the east-west gateway used in the multi-network and primary-remote configurations could also be used to enable access to the API server.

Environment Variables

The following environment variables will be used throughout to simplify the instructions:

VariableDescription
CTX_EXTERNAL_CLUSTERThe context name in the default Kubernetes configuration file used for accessing the external control plane cluster.
CTX_REMOTE_CLUSTERThe context name in the default Kubernetes configuration file used for accessing the remote cluster.
REMOTE_CLUSTER_NAMEThe name of the remote cluster.
EXTERNAL_ISTIOD_ADDRThe hostname for the ingress gateway on the external control plane cluster. This is used by the remote cluster to access the external control plane.
SSL_SECRET_NAMEThe name of the secret that holds the TLS certs for the ingress gateway on the external control plane cluster.

Set the CTX_EXTERNAL_CLUSTER, CTX_REMOTE_CLUSTER, and REMOTE_CLUSTER_NAME now. You will set the others later.

  1. $ export CTX_EXTERNAL_CLUSTER=external_cluster
  2. $ export CTX_REMOTE_CLUSTER=remote_cluster
  3. $ export REMOTE_CLUSTER_NAME=remote_cluster

Cluster configuration

Set up a gateway in the external cluster

Create the Istio install configuration for the ingress gateway that exposes the external control plane ports to other clusters:

  1. $ cat <<EOF > controlplane-gateway.yaml
  2. apiVersion: install.istio.io/v1alpha1
  3. kind: IstioOperator
  4. metadata:
  5.   namespace: istio-system
  6. spec:
  7.   components:
  8.     ingressGateways:
  9.       - name: istio-ingressgateway
  10.         enabled: true
  11.         k8s:
  12.           service:
  13.             ports:
  14.               - port: 15021
  15.                 targetPort: 15021
  16.                 name: status-port
  17.               - port: 15012
  18.                 targetPort: 15012
  19.                 name: tls-xds
  20.               - port: 15017
  21.                 targetPort: 15017
  22.                 name: tls-webhook
  23. EOF

Install the configuration to create the ingress gateway in the istio-system namespace of the external cluster:

  1. $ istioctl install -f controlplane-gateway.yaml --context="${CTX_EXTERNAL_CLUSTER}"

You may notice an istiod deployment created in the istio-system namespace. This is used only to configure the ingress gateway and is NOT the control plane used by remote clusters. This ingress gateway could, in fact, be configured to host multiple external control planes, in different namespaces on the cluster, even though in this example you will only deploy a single external istiod in the external-istiod namespace.

Configure your environment to expose the Istio ingress gateway service using a public hostname with TLS. Set the EXTERNAL_ISTIOD_ADDR environment variable to the hostname and SSL_SECRET_NAME environment variable to the secret that holds the TLS certs:

  1. $ export EXTERNAL_ISTIOD_ADDR=myexternal-istiod.cloud.com
  2. $ export SSL_SECRET_NAME=myexternal-istiod-secret

Create the Istio Gateway, VirtualService, and DestinationRule configuration for the yet to be installed external control plane:

  1. $ cat <<EOF > external-istiod-gw.yaml
  2. apiVersion: networking.istio.io/v1beta1
  3. kind: Gateway
  4. metadata:
  5.   name: external-istiod-gw
  6.   namespace: external-istiod
  7. spec:
  8.   selector:
  9.     istio: ingressgateway
  10.   servers:
  11.     - port:
  12.         number: 15012
  13.         protocol: https
  14.         name: https-XDS
  15.       tls:
  16.         mode: SIMPLE
  17.         credentialName: $SSL_SECRET_NAME
  18.       hosts:
  19.       - $EXTERNAL_ISTIOD_ADDR
  20.     - port:
  21.         number: 15017
  22.         protocol: https
  23.         name: https-WEBHOOK
  24.       tls:
  25.         mode: SIMPLE
  26.         credentialName: $SSL_SECRET_NAME
  27.       hosts:
  28.       - $EXTERNAL_ISTIOD_ADDR
  29. ---
  30. apiVersion: networking.istio.io/v1beta1
  31. kind: VirtualService
  32. metadata:
  33.    name: external-istiod-vs
  34.    namespace: external-istiod
  35. spec:
  36.     hosts:
  37.     - $EXTERNAL_ISTIOD_ADDR
  38.     gateways:
  39.     - external-istiod-gw
  40.     http:
  41.     - match:
  42.       - port: 15012
  43.       route:
  44.       - destination:
  45.           host: istiod.external-istiod.svc.cluster.local
  46.           port:
  47.             number: 15012
  48.     - match:
  49.       - port: 15017
  50.       route:
  51.       - destination:
  52.           host: istiod.external-istiod.svc.cluster.local
  53.           port:
  54.             number: 443
  55. ---
  56. apiVersion: networking.istio.io/v1alpha3
  57. kind: DestinationRule
  58. metadata:
  59.   name: external-istiod-dr
  60.   namespace: external-istiod
  61. spec:
  62.   host: istiod.external-istiod.svc.cluster.local
  63.   trafficPolicy:
  64.     portLevelSettings:
  65.     - port:
  66.         number: 15012
  67.       tls:
  68.         mode: SIMPLE
  69.       connectionPool:
  70.         http:
  71.           h2UpgradePolicy: UPGRADE
  72.     - port:
  73.         number: 443
  74.       tls:
  75.         mode: SIMPLE
  76. EOF

Create the external-istiod namespace and apply the configuration:

  1. $ kubectl create namespace external-istiod --context="${CTX_EXTERNAL_CLUSTER}"
  2. $ kubectl apply -f external-istiod-gw.yaml --context="${CTX_EXTERNAL_CLUSTER}"

Set up the remote cluster

Create the remote Istio install configuration:

  1. $ cat <<EOF > remote-config-cluster.yaml
  2. apiVersion: install.istio.io/v1alpha1
  3. kind: IstioOperator
  4. metadata:
  5.  namespace: external-istiod
  6. spec:
  7.   profile: remote
  8.   meshConfig:
  9.     rootNamespace: external-istiod
  10.     defaultConfig:
  11.       discoveryAddress: $EXTERNAL_ISTIOD_ADDR:15012
  12.       proxyMetadata:
  13.         XDS_ROOT_CA: /etc/ssl/certs/ca-certificates.crt
  14.         CA_ROOT_CA: /etc/ssl/certs/ca-certificates.crt
  15.   components:
  16.     pilot:
  17.       enabled: false
  18.     istiodRemote:
  19.       enabled: true
  21.   values:
  22.     global:
  23.       caAddress: $EXTERNAL_ISTIOD_ADDR:15012
  24.       istioNamespace: external-istiod
  25.       meshID: mesh1
  26.       multiCluster:
  27.         clusterName: $REMOTE_CLUSTER_NAME
  28.     istiodRemote:
  29.       injectionURL: https://$EXTERNAL_ISTIOD_ADDR:15017/inject
  30.     base:
  31.       validationURL: https://$EXTERNAL_ISTIOD_ADDR:15017/validate
  32. EOF

Install the configuration on the remote cluster:

  1. $ istioctl manifest generate -f remote-config-cluster.yaml  | kubectl apply --context="${CTX_REMOTE_CLUSTER}" -f -

NOTE: An ingress gateway, for accessing services in the remote cluster mesh, is included in the above installation. However it will not start working until you install the external control plane in the next section.

Set up the control plane in the external cluster

The control plane in the external cluster needs access to the remote cluster to discover services, endpoints, and pod attributes. Create a secret with credentials to access the remote cluster’s kube-apiserver and install it in the external cluster.

  1. $ kubectl create sa istiod-service-account -n external-istiod --context="${CTX_EXTERNAL_CLUSTER}"
  2. $ istioctl x create-remote-secret \
  3.   --context="${CTX_REMOTE_CLUSTER}" \
  4.   --type=config \
  5.   --namespace=external-istiod | \
  6.   kubectl apply -f - --context="${CTX_EXTERNAL_CLUSTER}"

Create the Istio install configuration to create the control plane in the external-istiod namespace of the external cluster:

  1. $ cat <<EOF > external-istiod.yaml
  2. apiVersion: install.istio.io/v1alpha1
  3. kind: IstioOperator
  4. metadata:
  5.   namespace: external-istiod
  6. spec:
  7.   meshConfig:
  8.     rootNamespace: external-istiod
  9.     defaultConfig:
  10.       discoveryAddress: $EXTERNAL_ISTIOD_ADDR:15012
  11.       proxyMetadata:
  12.         XDS_ROOT_CA: /etc/ssl/certs/ca-certificates.crt
  13.         CA_ROOT_CA: /etc/ssl/certs/ca-certificates.crt
  14.   components:
  15.     base:
  16.       enabled: false
  17.     ingressGateways:
  18.     - name: istio-ingressgateway
  19.       enabled: false
  20.   values:
  21.     global:
  22.       caAddress: $EXTERNAL_ISTIOD_ADDR:15012
  23.       istioNamespace: external-istiod
  24.       operatorManageWebhooks: true
  25.       meshID: mesh1
  26.       multiCluster:
  27.         clusterName: $REMOTE_CLUSTER_NAME
  28.     pilot:
  29.       env:
  30.         INJECTION_WEBHOOK_CONFIG_NAME: ""
  31.         VALIDATION_WEBHOOK_CONFIG_NAME: ""
  32. EOF

Apply the Istio configuration on the external cluster:

  1. $ istioctl install -f external-istiod.yaml --context="${CTX_EXTERNAL_CLUSTER}"

Validate the installation

Confirm that the Istio ingress gateway is now running on the remote cluster.

  1. $ kubectl get pod -l app=istio-ingressgateway -n external-istiod --context="${CTX_REMOTE_CLUSTER}"

Deploy the helloworld sample to the remote cluster. Wait a few seconds for the helloworld pods to be running with sidecars injected.

  1. $ kubectl label namespace default istio-injection=enabled --context="${CTX_REMOTE_CLUSTER}"
  2. $ kubectl apply -f samples/helloworld/helloworld.yaml --context="${CTX_REMOTE_CLUSTER}"
  3. $ kubectl get pod -l app=helloworld --context="${CTX_REMOTE_CLUSTER}"

Expose the helloworld application on the ingress gateway:

  1. $ kubectl apply -f samples/helloworld/helloworld-gateway.yaml --context="${CTX_REMOTE_CLUSTER}"

Follow these instructions to set GATEWAY_URL and then confirm you can access the helloworld application:

  1. $ curl -s "http://${GATEWAY_URL}/hello" | grep -o "Hello"

Congratulations! You successfully installed an external control plane and used it to manage services running in a remote cluster!

See also

Deploying Istio Control Planes Outside the Mesh

A new deployment model for Istio.

Introducing istiod: simplifying the control plane

Istiod consolidates the Istio control plane components into a single binary.

Observability

Describes the telemetry and monitoring features provided by Istio.