Hubble Configuration

This page provides guidance to configure Hubble in a way that suits your environment. Instructions to enable Hubble are provided as part of each Cilium Installation guide.

TLS certificates

When Hubble Relay is deployed, Hubble listens on a TCP port on the host network. This allows Hubble Relay to communicate with all Hubble instances in the cluster. Connections between Hubble instances and Hubble Relay are secured using mutual TLS (mTLS) by default.

TLS certificates can be provided by manually on the Helm install command (user provided) or generate automatically via either:

User provided certificates

In order to use custom TLS certificates, hubble.tls.auto.enabled must be set to false and TLS certificates manually provided. This can be done by specifying the options below to Helm at install or upgrade time.

  1. --set hubble.tls.auto.enabled=false                          # disable automatic TLS certificate generation
  2. --set-file hubble.tls.ca.cert=ca.crt.b64                     # certificate of the CA that signs all certificates
  3. --set-file hubble.tls.server.cert=server.crt.b64             # certificate for Hubble server
  4. --set-file hubble.tls.server.key=server.key.b64              # private key for the Hubble server certificate
  5. --set-file hubble.relay.tls.client.cert=relay-client.crt.b64 # client certificate for Hubble Relay to connect to Hubble instances
  6. --set-file hubble.relay.tls.client.key=relay-client.key.b64  # private key for Hubble Relay client certificate
  7. --set-file hubble.relay.tls.server.cert=relay-server.crt.b64 # server certificate for Hubble Relay
  8. --set-file hubble.relay.tls.server.key=relay-server.key.b64  # private key for Hubble Relay server certificate
  9. --set-file hubble.ui.tls.client.cert=ui-client.crt.b64       # client certificate for Hubble UI
  10. --set-file hubble.ui.tls.client.key=ui-client.key.b64        # private key for Hubble UI client certificate

Options hubble.relay.tls.server.cert, hubble.relay.tls.server.key hubble.ui.tls.client.cert and hubble.ui.tls.client.key only need to be provided when hubble.relay.tls.server.enabled=true (default false) which enable TLS for the Hubble Relay server.

Note

Provided files must be base64 encoded PEM certificates.

In addition, the Common Name (CN) and Subject Alternative Name (SAN) of the certificate for Hubble server MUST be set to *.{cluster-name}.hubble-grpc.cilium.io where {cluster-name} is the cluster name defined by cluster.name (defaults to default).

Auto generated certificates via Helm

When using Helm, TLS certificates are (re-)generated every time Helm is used for install or upgrade. As Hubble server and Hubble Relay support TLS certificates hot reloading, including CA certificates, this does not disrupt any existing connection. New connections are automatically established using the new certificates without having to restart Hubble server or Hubble Relay.

  1. --set hubble.tls.auto.enabled=true               # enable automatic TLS certificate generation
  2. --set hubble.tls.auto.method=helm                # auto generate certificates using helm method
  3. --set hubble.tls.auto.certValidityDuration=1095  # certificates validity duration in days (default 3 years)

The downside of the Helm method is that while certificates are automatically generated, they are not automatically renewed. Consequently, running helm upgrade is required when certificates are about to expire (i.e. before the configured hubble.tls.auto.certValidityDuration).

Auto generated certificates via certgen

Like the Helm method, certgen generates the TLS certificates at installation time and a Kubernetes CronJob is scheduled to renew them (regardless of their expiration date).

  1. --set hubble.tls.auto.enabled=true               # enable automatic TLS certificate generation
  2. --set hubble.tls.auto.method=cronJob             # auto generate certificates using cronJob method
  3. --set hubble.tls.auto.certValidityDuration=1095  # certificates validity duration in days (default 3 years)
  4. --set hubble.tls.auto.schedule="0 0 1 */4 *"     # schedule for certificates re-generation (crontab syntax)

Auto generated certificates via cert-manager

This method relies on cert-manager to generate the TLS certificates. cert-manager has becomes the de facto way to manage TLS on Kubernetes, and it has the following advantages compared to the previously documented methods:

Installation steps:

  1. First, install cert-manager and setup an issuer. Please make sure that your issuer is be able to create certificates under the cilium.io domain name.

  2. Install/upgrade Cilium including the following Helm flags:

  1. --set hubble.tls.auto.enabled=true               # enable automatic TLS certificate generation
  2. --set hubble.tls.auto.method=certmanager         # auto generate certificates using cert-manager
  3. --set hubble.tls.auto.certValidityDuration=1095  # certificates validity duration in days (default 3 years)
  4. --set hubble.tls.auto.certManagerIssuerRef.group="cert-manager.io" # Reference to cert-manager's issuer
  5. --set hubble.tls.auto.certManagerIssuerRef.kind="ClusterIssuer"
  6. --set hubble.tls.auto.certManagerIssuerRef.name="ca-issuer"

Troubleshooting:

While installing Cilium or cert-manager you may get the following error:

  1. Error: Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp x.x.x.x:443: connect: connection refused

This happens when cert-manager’s webhook (which is used to verify the Certificate’s CRD resources) is not available. There are several ways to resolve this issue. Pick one of the options below:

Install CRDs firstUpgrade CiliumDisable webhookHost network webhook

Install cert-manager CRDs before Cilium and cert-manager (see cert-manager’s documentation about installing CRDs with kubectl):

  1. $ kubectl create -f cert-manager.crds.yaml

Then install cert-manager, configure an issuer, and install Cilium.

Upgrade Cilium from an installation with TLS disabled:

  1. $ helm install cilium cilium/cilium \
  2.     --set hubble.tls.enabled=false \
  3.     ...

Then install cert-manager, configure an issuer, and upgrade Cilium enabling TLS:

  1. $ helm install cilium cilium/cilium --set hubble.tls.enabled=true

Disable cert-manager validation (assuming Cilium is installed in the kube-system namespace):

  1. $ kubectl label namespace kube-system cert-manager.io/disable-validation=true

Then install Cilium, cert-manager, and configure an issuer.

Configure cert-manager to expose its webhook within the host network namespace:

  1. $ helm install cert-manager jetstack/cert-manager \
  2.         --set webhook.hostNetwork=true \
  3.         --set webhook.tolerations='["operator": "Exists"]'

Then configure an issuer and install Cilium.