Connect Service Mesh on Kubernetes

Connect is a feature built into to Consul that enables automatic service-to-service authorization and connection encryption across your Consul services. Connect can be used with Kubernetes to secure pod communication with other pods and external Kubernetes services.

The Connect sidecar running Envoy can be automatically injected into pods in your cluster, making configuration for Kubernetes automatic. This functionality is provided by the consul-k8s project and can be automatically installed and configured using the Consul Helm chart.

Usage

When the Connect injector is installed, the Connect sidecar can be automatically added to all pods. This sidecar can both accept and establish connections using Connect, enabling the pod to communicate to clients and dependencies exclusively over authorized and encrypted connections.

Note: The pod specifications in this section are valid and use publicly available images. If you’ve installed the Connect injector, feel free to run the pod specifications in this section to try Connect with Kubernetes. Please note the documentation below this section on how to properly install and configure the Connect injector.

Accepting Inbound Connections

An example pod is shown below with Connect enabled to accept inbound connections. Notice that the pod would still be fully functional without Connect. Minimal to zero modifications are required to pod specifications to enable Connect in Kubernetes.

This pod specification starts a server that responds to any HTTP request with the static text “hello world”.

  1. apiVersion: v1
  2. kind: ServiceAccount
  3. metadata:
  4.   name: static-server
  5. ---
  6. apiVersion: v1
  7. kind: Pod
  8. metadata:
  9.   name: static-server
  10.   annotations:
  11.     'consul.hashicorp.com/connect-inject': 'true'
  12. spec:
  13.   containers:
  14.     # This name will be the service name in Consul.
  15.     - name: static-server
  16.       image: hashicorp/http-echo:latest
  17.       args:
  18.         - -text="hello world"
  19.         - -listen=:8080
  20.       ports:
  21.         - containerPort: 8080
  22.           name: http
  23.     # If ACLs are enabled, the serviceAccountName must match the Consul service name.
  24.   serviceAccountName: static-server

The only change for Connect is the addition of the consul.hashicorp.com/connect-inject annotation. This enables injection for this pod. The injector can also be configured to automatically inject unless explicitly disabled, but the default installation requires opt-in using the annotation shown above.

This will start a Connect sidecar that listens on a random port registered with Consul and proxies valid inbound connections to port 8080 in the pod. To establish a connection to the pod using Connect, a client must use another Connect proxy. The client Connect proxy will use Consul service discovery to find all available upstream proxies and their public ports.

In the example above, the server is listening on :8080. This means the server will still bind to the pod IP and allow external connections. This is useful to transition to Connect by allowing both Connect and non-Connect connections. To restrict access to only Connect-authorized clients, any listeners should bind to localhost only (such as 127.0.0.1).

The service name registered in Consul will be set to the name of the first container in the Pod. This can be customized with the consul.hashicorp.com/connect-service annotation. If using ACLs, this name must be the same as the Pod’s ServiceAccount name.

Connecting to Connect-Enabled Services

The example pod specification below configures a pod that is capable of establishing connections to our previous example “static-server” service. The connection to this static text service happens over an authorized and encrypted connection via Connect.

  1. apiVersion: v1
  2. kind: ServiceAccount
  3. metadata:
  4.   name: static-client
  5. ---
  6. apiVersion: v1
  7. kind: Pod
  8. metadata:
  9.   name: static-client
  10.   annotations:
  11.     'consul.hashicorp.com/connect-inject': 'true'
  12.     'consul.hashicorp.com/connect-service-upstreams': 'static-server:1234'
  13. spec:
  14.   containers:
  15.     # This name will be the service name in Consul.
  16.     - name: static-client
  17.       image: tutum/curl:latest
  18.       # Just spin & wait forever, we'll use `kubectl exec` to demo
  19.       command: ['/bin/sh', '-c', '--']
  20.       args: ['while true; do sleep 30; done;']
  21.     # If ACLs are enabled, the serviceAccountName must match the Consul service name.
  22.   serviceAccountName: static-client

Pods must specify upstream dependencies with the consul.hashicorp.com/connect-service-upstreams annotation. This annotation declares the names of any upstream dependencies and a local port for the proxy to listen on. When a connection is established to that local port, the proxy establishes a connection to the target service (static-server in this example) using mutual TLS and identifying as the source service (static-client in this example).

The injector will also set environment variables <NAME>_CONNECT_SERVICE_HOST and <NAME>_CONNECT_SERVICE_PORT in every container in the pod for every defined upstream. This is analogous to the standard Kubernetes service environment variables, but point instead to the correct local proxy port to establish connections via Connect.

Any containers running in the pod that need to establish connections to dependencies must be reconfigured to use the local upstream address either directly or using the environment variables set by the injector (defined above). This means pods should not use Kubernetes service DNS or environment variables for these connections.

We can verify access to the static text server using kubectl exec. Notice that we use the local address and port from the upstream annotation (1234) for this verification.

  1. $ kubectl exec static-client -- curl -s http://127.0.0.1:1234/
  2. "hello world"

We can control access to the server using intentions. If you use the Consul UI or CLI to create a deny intention between “static-client” and “static-server”, connections are immediately rejected without updating either of the running pods. You can then remove this intention to allow connections again.

  1. $ kubectl exec static-client -- curl -s http://127.0.0.1:1234/
  2. command terminated with exit code 52

Available Annotations

Annotations can be used to configure the injection behavior.

Deployments, StatefulSets, etc.

The annotations for configuring Connect must be on the pod specification. Since higher level resources such as Deployments wrap pod specification templates, Connect can be used with all of these higher level constructs, too.

An example Deployment below shows how to enable Connect injection:

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: consul-example-deployment
  5. spec:
  6.   replicas: 1
  7.   selector:
  8.     matchLabels:
  9.       app: consul-example
  10.   template:
  11.     metadata:
  12.       labels:
  13.         app: consul-example
  14.       annotations:
  15.         'consul.hashicorp.com/connect-inject': 'true'
  16.     spec:
  17.       containers:
  18.         - name: consul-example
  19.           image: 'nginx'
  20.       serviceAccountName: consul-example
  21. ---
  22. apiVersion: v1
  23. kind: ServiceAccount
  24. metadata:
  25.   name: consul-example

A common mistake is to set the annotation on the Deployment or other resource. Ensure that the injector annotations are specified on the pod specification template as shown above.

Installation and Configuration

The Connect sidecar proxy is injected via a mutating admission webhook provided by the consul-k8s project. This enables the automatic pod mutation shown in the usage section above. Installation of the mutating admission webhook is automated using the Helm chart.

To install the Connect injector, enable the Connect injection feature using Helm values and upgrade the installation using helm upgrade for existing installs or helm install for a fresh install.

  1. connectInject:
  2.   enabled: true
  4. controller:
  5.   enabled: true

This will configure the injector to inject when the injection annotation is set to true. Other values in the Helm chart can be used to limit the namespaces the injector runs in, enable injection by default, and more.

Controlling Injection Via Annotation

By default, the injector will inject only when the injection annotation on the pod (not the deployment) is set to true:

  1. annotations:
  2.   'consul.hashicorp.com/connect-inject': 'true'

Injection Defaults

If you wish for the injector to always inject, you can set the default to true in the Helm chart:

  1. connectInject:
  2.   enabled: true
  3.   default: true

You can then exclude specific pods via annotation:

  1. annotations:
  2.   'consul.hashicorp.com/connect-inject': 'false'

Controlling Injection Via Namespace

You can control which Kubernetes namespaces are allowed to be injected via the k8sAllowNamespaces and k8sDenyNamespaces keys:

  1. connectInject:
  2.   enabled: true
  3.   k8sAllowNamespaces: ['*']
  4.   k8sDenyNamespaces: []

In the default configuration (shown above), services from all namespaces are allowed to be injected. Whether or not they’re injected depends on the value of connectInject.default and the consul.hashicorp.com/connect-inject annotation.

If you wish to only enable injection in specific namespaces, you can list only those namespaces in the k8sAllowNamespaces key. In the configuration below only the my-ns-1 and my-ns-2 namespaces will be enabled for injection. All other namespaces will be ignored, even if the connect inject annotation is set.

  1. connectInject:
  2.   enabled: true
  3.   k8sAllowNamespaces: ['my-ns-1', 'my-ns-2']
  4.   k8sDenyNamespaces: []

If you wish to enable injection in every namespace except specific namespaces, you can use * in the allow list to allow all namespaces and then specify the namespaces to exclude in the deny list:

  1. connectInject:
  2.   enabled: true
  3.   k8sAllowNamespaces: ['*']
  4.   k8sDenyNamespaces: ['no-inject-ns-1', 'no-inject-ns-2']

NOTE: The deny list takes precedence over the allow list. If a namespace is listed in both lists, it will not be synced.

NOTE: The kube-system and kube-public namespaces will never be injected.

Consul Enterprise Namespaces

Consul Enterprise 1.7+ supports Consul namespaces. When Kubernetes pods are registered into Consul, you can control which Consul namespace they are registered into.

There are three options available:

  1. Single Destination Namespace – Register all Kubernetes pods, regardless of namespace, into the same Consul namespace.

    This can be configured with:

    1. global:
    2.   enableConsulNamespaces: true
    4. connectInject:
    5.   enabled: true
    6.   consulNamespaces:
    7.     consulDestinationNamespace: 'my-consul-ns'

    NOTE: If the destination namespace does not exist we will create it.

  2. Mirror Namespaces - Register each Kubernetes pod into a Consul namespace with the same name as its Kubernetes namespace. For example, pod foo in Kubernetes namespace ns-1 will be synced to the Consul namespace ns-1. If a mirrored namespace does not exist in Consul, it will be created.

    This can be configured with:

    1. global:
    2.   enableConsulNamespaces: true
    4. connectInject:
    5.   enabled: true
    6.   consulNamespaces:
    7.     mirroringK8S: true

  3. Mirror Namespaces With Prefix - Register each Kubernetes pod into a Consul namespace with the same name as its Kubernetes namespace with a prefix. For example, given a prefix k8s-, pod foo in Kubernetes namespace ns-1 will be synced to the Consul namespace k8s-ns-1.

    This can be configured with:

    1. global:
    2.   enableConsulNamespaces: true
    4. connectInject:
    5.   enabled: true
    6.   consulNamespaces:
    7.     mirroringK8S: true
    8.     mirroringK8SPrefix: 'k8s-'

Consul Enterprise Namespace Upstreams

To specify the namespace of your upstream services in the upstream annotation, use the format [service-name].[namespace]:[port]:[optional datacenter]:

  1. annotations:
  2.   'consul.hashicorp.com/connect-inject': 'true'
  3.   'consul.hashicorp.com/connect-service-upstreams': '[service-name].[namespace]:[port]:[optional datacenter]'

See consul.hashicorp.com/connect-service-upstreams for more details.

Verifying the Installation

To verify the installation, run the “Accepting Inbound Connections” example from the “Usage” section above. After running this example, run kubectl get pod static-server -o yaml. In the raw YAML output, you should see injected Connect containers and an annotation consul.hashicorp.com/connect-inject-status set to injected. This confirms that injection is working properly.

If you do not see this, then use kubectl logs against the injector pod and note any errors.