Custom CA Integration using Kubernetes CSR

This feature is actively in development and is considered experimental.

This feature requires Kubernetes version >= 1.18.

This task shows how to provision workload certificates using a custom certificate authority that integrates with the Kubernetes CSR API. Different workloads can get their certificates signed from different cert-signers. Each cert-signer is effectively a different CA. It is expected that workloads whose certificates are issued from the same cert-signer can talk mTLS to each other while workloads signed by different signers cannot. This feature leverages Chiron, a lightweight component linked with Istiod that signs certificates using the Kubernetes CSR API.

For this example, we use open-source cert-manager. Cert-manager has added experimental Support for Kubernetes CertificateSigningRequests starting with version 1.4.

Deploy custom CA controller in the Kubernetes cluster

  1. Deploy cert-manager according to the installation doc.

    Make sure to enable feature gate: --feature-gates=ExperimentalCertificateSigningRequestControllers=true

    1. $ helm repo add jetstack https://charts.jetstack.io
    2. $ helm repo update
    3. $ helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --set featureGates="ExperimentalCertificateSigningRequestControllers=true" --set installCRDs=true

  2. Create three self signed cluster issuers istio-system, foo and bar for cert-manager. Note: Namespace issuers and other types of issuers can also be used.

    1. $ cat <<EOF > ./selfsigned-issuer.yaml
    2. apiVersion: cert-manager.io/v1
    3. kind: ClusterIssuer
    4. metadata:
    5.   name: selfsigned-bar-issuer
    6. spec:
    7.   selfSigned: {}
    8. ---
    9. apiVersion: cert-manager.io/v1
    10. kind: Certificate
    11. metadata:
    12.   name: bar-ca
    13.   namespace: cert-manager
    14. spec:
    15.   isCA: true
    16.   commonName: bar
    17.   secretName: bar-ca-selfsigned
    18.   issuerRef:
    19.     name: selfsigned-bar-issuer
    20.     kind: ClusterIssuer
    21.     group: cert-manager.io
    22. ---
    23. apiVersion: cert-manager.io/v1
    24. kind: ClusterIssuer
    25. metadata:
    26.   name: bar
    27. spec:
    28.   ca:
    29.     secretName: bar-ca-selfsigned
    30. ---
    31. apiVersion: cert-manager.io/v1
    32. kind: ClusterIssuer
    33. metadata:
    34.   name: selfsigned-foo-issuer
    35. spec:
    36.   selfSigned: {}
    37. ---
    38. apiVersion: cert-manager.io/v1
    39. kind: Certificate
    40. metadata:
    41.   name: foo-ca
    42.   namespace: cert-manager
    43. spec:
    44.   isCA: true
    45.   commonName: foo
    46.   secretName: foo-ca-selfsigned
    47.   issuerRef:
    48.     name: selfsigned-foo-issuer
    49.     kind: ClusterIssuer
    50.     group: cert-manager.io
    51. ---
    52. apiVersion: cert-manager.io/v1
    53. kind: ClusterIssuer
    54. metadata:
    55.   name: foo
    56. spec:
    57.   ca:
    58.     secretName: foo-ca-selfsigned
    59. ---
    60. apiVersion: cert-manager.io/v1
    61. kind: ClusterIssuer
    62. metadata:
    63.   name: selfsigned-istio-issuer
    64. spec:
    65.   selfSigned: {}
    66. ---
    67. apiVersion: cert-manager.io/v1
    68. kind: Certificate
    69. metadata:
    70.   name: istio-ca
    71.   namespace: cert-manager
    72. spec:
    73.   isCA: true
    74.   commonName: istio-system
    75.   secretName: istio-ca-selfsigned
    76.   issuerRef:
    77.     name: selfsigned-istio-issuer
    78.     kind: ClusterIssuer
    79.     group: cert-manager.io
    80. ---
    81. apiVersion: cert-manager.io/v1
    82. kind: ClusterIssuer
    83. metadata:
    84.   name: istio-system
    85. spec:
    86.   ca:
    87.     secretName: istio-ca-selfsigned
    88. EOF
    89. $ kubectl apply -f ./selfsigned-issuer.yaml

Verify secrets are created for each cluster issuer

  1. $ kubectl get secret -n cert-manager -l controller.cert-manager.io/fao=true
  2. NAME                  TYPE                DATA   AGE
  3. bar-ca-selfsigned     kubernetes.io/tls   3      3m36s
  4. foo-ca-selfsigned     kubernetes.io/tls   3      3m36s
  5. istio-ca-selfsigned   kubernetes.io/tls   3      3m38s

Export root certificates for each cluster issuer

  1. $ export ISTIOCA=$(kubectl get clusterissuers istio-system -o jsonpath='{.spec.ca.secretName}' | xargs kubectl get secret -n cert-manager -o jsonpath='{.data.ca\.crt}' | base64 -d | sed 's/^/        /')
  2. $ export FOOCA=$(kubectl get clusterissuers foo -o jsonpath='{.spec.ca.secretName}' | xargs kubectl get secret -n cert-manager -o jsonpath='{.data.ca\.crt}' | base64 -d | sed 's/^/        /')
  3. $ export BARCA=$(kubectl get clusterissuers bar -o jsonpath='{.spec.ca.secretName}' | xargs kubectl get secret -n cert-manager -o jsonpath='{.data.ca\.crt}' | base64 -d | sed 's/^/        /')

Deploy Istio with default cert-signer info

  1. Deploy Istio on the cluster using istioctl with the following configuration. The ISTIO_META_CERT_SIGNER is the default cert-signer for workloads.

    1. $ cat <<EOF > ./istio.yaml
    2. apiVersion: install.istio.io/v1alpha1
    3. kind: IstioOperator
    4. spec:
    5.   values:
    6.     pilot:
    7.       env:
    8.         EXTERNAL_CA: ISTIOD_RA_KUBERNETES_API
    9.   meshConfig:
    10.     defaultConfig:
    11.       proxyMetadata:
    12.         ISTIO_META_CERT_SIGNER: istio-system
    13.     caCertificates:
    14.     - pem: |
    15. $ISTIOCA
    16.       certSigners:
    17.       - clusterissuers.cert-manager.io/istio-system
    18.     - pem: |
    19. $FOOCA
    20.       certSigners:
    21.       - clusterissuers.cert-manager.io/foo
    22.     - pem: |
    23. $BARCA
    24.       certSigners:
    25.       - clusterissuers.cert-manager.io/bar
    26.   components:
    27.     pilot:
    28.       k8s:
    29.         env:
    30.         - name: CERT_SIGNER_DOMAIN
    31.           value: clusterissuers.cert-manager.io
    32.         - name: PILOT_CERT_PROVIDER
    33.           value: k8s.io/clusterissuers.cert-manager.io/istio-system
    34.         overlays:
    35.           - kind: ClusterRole
    36.             name: istiod-clusterrole-istio-system
    37.             patches:
    38.               - path: rules[-1]
    39.                 value: |
    40.                   apiGroups:
    41.                   - certificates.k8s.io
    42.                   resourceNames:
    43.                   - clusterissuers.cert-manager.io/foo
    44.                   - clusterissuers.cert-manager.io/bar
    45.                   - clusterissuers.cert-manager.io/istio-system
    46.                   resources:
    47.                   - signers
    48.                   verbs:
    49.                   - approve
    50. EOF
    51. $ istioctl install --skip-confirmation -f ./istio.yaml

  2. Create the bar and foo namespaces.

    1. $ kubectl create ns bar
    2. $ kubectl create ns foo

  3. Deploy the proxyconfig-bar.yaml in the bar namespace to define cert-signer for workloads in the bar namespace.

    1. $ cat <<EOF > ./proxyconfig-bar.yaml
    2. apiVersion: networking.istio.io/v1beta1
    3. kind: ProxyConfig
    4. metadata:
    5.   name: barpc
    6.   namespace: bar
    7. spec:
    8.   environmentVariables:
    9.     ISTIO_META_CERT_SIGNER: bar
    10. EOF
    11. $ kubectl apply  -f ./proxyconfig-bar.yaml

  4. Deploy the proxyconfig-foo.yaml in the foo namespace to define cert-signer for workloads in the foo namespace.

    1. $ cat <<EOF > ./proxyconfig-foo.yaml
    2. apiVersion: networking.istio.io/v1beta1
    3. kind: ProxyConfig
    4. metadata:
    5.   name: foopc
    6.   namespace: foo
    7. spec:
    8.   environmentVariables:
    9.     ISTIO_META_CERT_SIGNER: foo
    10. EOF
    11. $ kubectl apply  -f ./proxyconfig-foo.yaml

  5. Deploy the httpbin and curl sample applications in the foo and bar namespaces.

    1. $ kubectl label ns foo istio-injection=enabled
    2. $ kubectl label ns bar istio-injection=enabled
    3. $ kubectl apply -f samples/httpbin/httpbin.yaml -n foo
    4. $ kubectl apply -f samples/curl/curl.yaml -n foo
    5. $ kubectl apply -f samples/httpbin/httpbin.yaml -n bar

Verify the network connectivity between httpbin and curl within the same namespace

When the workloads are deployed, they send CSR requests with related signer info. Istiod forwards the CSR request to the custom CA for signing. The custom CA will use the correct cluster issuer to sign the cert back. Workloads under foo namespace will use foo cluster issuers while workloads under bar namespace will use the bar cluster issuers. To verify that they have indeed been signed by correct cluster issuers, we can verify workloads under the same namespace can communicate while workloads under the different namespace cannot communicate.

  1. Set the CURL_POD_FOO environment variable to the name of curl pod.

    1. $ export CURL_POD_FOO=$(kubectl get pod -n foo -l app=curl -o jsonpath={.items..metadata.name})

  2. Check network connectivity between service curl and httpbin in the foo namespace.

    1. $ kubectl exec "$CURL_POD_FOO" -n foo -c curl -- curl http://httpbin.foo:8000/html
    2. <!DOCTYPE html>
    3. <html>
    4.   <head>
    5.   </head>
    6.   <body>
    7.       <h1>Herman Melville - Moby-Dick</h1>
    9.       <div>
    10.         <p>
    11.           Availing himself of the mild...
    12.         </p>
    13.       </div>
    14.   </body>

  3. Check network connectivity between service curl in the foo namespace and httpbin in the bar namespace.

    1. $ kubectl exec "$CURL_POD_FOO" -n foo -c curl -- curl http://httpbin.bar:8000/html
    2. upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

Cleanup

  • Remove the namespaces and uninstall Istio and cert-manager:

    1. $ kubectl delete ns foo
    2. $ kubectl delete ns bar
    3. $ istioctl uninstall --purge -y
    4. $ helm delete -n cert-manager cert-manager
    5. $ kubectl delete ns istio-system cert-manager
    6. $ unset ISTIOCA FOOCA BARCA
    7. $ rm -rf istio.yaml proxyconfig-foo.yaml proxyconfig-bar.yaml selfsigned-issuer.yaml

Reasons to use this feature

  • Custom CA Integration - By specifying a Signer name in the Kubernetes CSR Request, this feature allows Istio to integrate with custom Certificate Authorities using the Kubernetes CSR API interface. This does require the custom CA to implement a Kubernetes controller to watch the CertificateSigningRequest Resources and act on them.

  • Better multi-tenancy - By specifying a different cert-signer for different workloads, certificates for different tenant’s workloads can be signed by different CAs.