- Custom CA Integration using Kubernetes CSR
- Deploy custom CA controller in the Kubernetes cluster
- Verify secrets are created for each cluster issuer
- Export root certificates for each cluster issuer
- Deploy Istio with default cert-signer info
- Verify the network connectivity between
httpbin
andsleep
within the same namespace - Cleanup
- Reasons to use this feature
Custom CA Integration using Kubernetes CSR
This feature is actively in development and is considered experimental.
This feature requires Kubernetes version >= 1.18.
This task shows how to provision workload certificates using a custom certificate authority that integrates with the Kubernetes CSR API. Different workloads can get their certificates signed from different cert-signers. Each cert-signer is effectively a different CA. It is expected that workloads whose certificates are issued from the same cert-signer can talk mTLS to each other while workloads signed by different signers cannot. This feature leverages Chiron, a lightweight component linked with Istiod that signs certificates using the Kubernetes CSR API.
For this example, we use open-source cert-manager. Cert-manager has added experimental Support for Kubernetes CertificateSigningRequests starting with version 1.4.
Deploy custom CA controller in the Kubernetes cluster
Deploy cert-manager according to the installation doc.
Make sure to enable feature gate:
--feature-gates=ExperimentalCertificateSigningRequestControllers=true
$ helm repo add jetstack https://charts.jetstack.io
$ helm repo update
$ helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --set featureGates="ExperimentalCertificateSigningRequestControllers=true" --set installCRDs=true
Create three self signed cluster issuers
istio-system
,foo
andbar
for cert-manager. Note: Namespace issuers and other types of issuers can also be used.$ cat <<EOF > ./selfsigned-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: selfsigned-bar-issuer
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: bar-ca
namespace: cert-manager
spec:
isCA: true
commonName: bar
secretName: bar-ca-selfsigned
issuerRef:
name: selfsigned-bar-issuer
kind: ClusterIssuer
group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: bar
spec:
ca:
secretName: bar-ca-selfsigned
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: selfsigned-foo-issuer
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: foo-ca
namespace: cert-manager
spec:
isCA: true
commonName: foo
secretName: foo-ca-selfsigned
issuerRef:
name: selfsigned-foo-issuer
kind: ClusterIssuer
group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: foo
spec:
ca:
secretName: foo-ca-selfsigned
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: selfsigned-istio-issuer
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: istio-ca
namespace: cert-manager
spec:
isCA: true
commonName: istio-system
secretName: istio-ca-selfsigned
issuerRef:
name: selfsigned-istio-issuer
kind: ClusterIssuer
group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: istio-system
spec:
ca:
secretName: istio-ca-selfsigned
EOF
$ kubectl apply -f ./selfsigned-issuer.yaml
Verify secrets are created for each cluster issuer
$ kubectl get secret -n cert-manager -l controller.cert-manager.io/fao=true
NAME TYPE DATA AGE
bar-ca-selfsigned kubernetes.io/tls 3 3m36s
foo-ca-selfsigned kubernetes.io/tls 3 3m36s
istio-ca-selfsigned kubernetes.io/tls 3 3m38s
Export root certificates for each cluster issuer
$ export ISTIOCA=$(kubectl get clusterissuers istio-system -o jsonpath='{.spec.ca.secretName}' | xargs kubectl get secret -n cert-manager -o jsonpath='{.data.ca\.crt}' | base64 -d | sed 's/^/ /')
$ export FOOCA=$(kubectl get clusterissuers foo -o jsonpath='{.spec.ca.secretName}' | xargs kubectl get secret -n cert-manager -o jsonpath='{.data.ca\.crt}' | base64 -d | sed 's/^/ /')
$ export BARCA=$(kubectl get clusterissuers bar -o jsonpath='{.spec.ca.secretName}' | xargs kubectl get secret -n cert-manager -o jsonpath='{.data.ca\.crt}' | base64 -d | sed 's/^/ /')
Deploy Istio with default cert-signer info
Deploy Istio on the cluster using
istioctl
with the following configuration. TheISTIO_META_CERT_SIGNER
is the default cert-signer for workloads.$ cat <<EOF > ./istio.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
values:
pilot:
env:
EXTERNAL_CA: ISTIOD_RA_KUBERNETES_API
meshConfig:
defaultConfig:
proxyMetadata:
ISTIO_META_CERT_SIGNER: istio-system
caCertificates:
- pem: |
$ISTIOCA
certSigners:
- clusterissuers.cert-manager.io/istio-system
- pem: |
$FOOCA
certSigners:
- clusterissuers.cert-manager.io/foo
- pem: |
$BARCA
certSigners:
- clusterissuers.cert-manager.io/bar
components:
pilot:
k8s:
env:
- name: CERT_SIGNER_DOMAIN
value: clusterissuers.cert-manager.io
- name: PILOT_CERT_PROVIDER
value: k8s.io/clusterissuers.cert-manager.io/istio-system
overlays:
- kind: ClusterRole
name: istiod-clusterrole-istio-system
patches:
- path: rules[-1]
value: |
apiGroups:
- certificates.k8s.io
resourceNames:
- clusterissuers.cert-manager.io/foo
- clusterissuers.cert-manager.io/bar
- clusterissuers.cert-manager.io/istio-system
resources:
- signers
verbs:
- approve
EOF
$ istioctl install --skip-confirmation -f ./istio.yaml
Create the
bar
andfoo
namespaces.$ kubectl create ns bar
$ kubectl create ns foo
Deploy the
proxyconfig-bar.yaml
in thebar
namespace to define cert-signer for workloads in thebar
namespace.$ cat <<EOF > ./proxyconfig-bar.yaml
apiVersion: networking.istio.io/v1beta1
kind: ProxyConfig
metadata:
name: barpc
namespace: bar
spec:
environmentVariables:
ISTIO_META_CERT_SIGNER: bar
EOF
$ kubectl apply -f ./proxyconfig-bar.yaml
Deploy the
proxyconfig-foo.yaml
in thefoo
namespace to define cert-signer for workloads in thefoo
namespace.$ cat <<EOF > ./proxyconfig-foo.yaml
apiVersion: networking.istio.io/v1beta1
kind: ProxyConfig
metadata:
name: foopc
namespace: foo
spec:
environmentVariables:
ISTIO_META_CERT_SIGNER: foo
EOF
$ kubectl apply -f ./proxyconfig-foo.yaml
Deploy the
httpbin
andsleep
sample applications in thefoo
andbar
namespaces.$ kubectl label ns foo istio-injection=enabled
$ kubectl label ns bar istio-injection=enabled
$ kubectl apply -f samples/httpbin/httpbin.yaml -n foo
$ kubectl apply -f samples/sleep/sleep.yaml -n foo
$ kubectl apply -f samples/httpbin/httpbin.yaml -n bar
Verify the network connectivity between httpbin
and sleep
within the same namespace
When the workloads are deployed, they send CSR requests with related signer info. Istiod forwards the CSR request to the custom CA for signing. The custom CA will use the correct cluster issuer to sign the cert back. Workloads under foo
namespace will use foo
cluster issuers while workloads under bar
namespace will use the bar
cluster issuers. To verify that they have indeed been signed by correct cluster issuers, we can verify workloads under the same namespace can communicate while workloads under the different namespace cannot communicate.
Set the
SLEEP_POD_FOO
environment variable to the name ofsleep
pod.$ export SLEEP_POD_FOO=$(kubectl get pod -n foo -l app=sleep -o jsonpath={.items..metadata.name})
Check network connectivity between service
sleep
andhttpbin
in thefoo
namespace.$ kubectl exec "$SLEEP_POD_FOO" -n foo -c sleep -- curl http://httpbin.foo:8000/html
<!DOCTYPE html>
<html>
<head>
</head>
<body>
<h1>Herman Melville - Moby-Dick</h1>
<div>
<p>
Availing himself of the mild...
</p>
</div>
</body>
Check network connectivity between service
sleep
in thefoo
namespace andhttpbin
in thebar
namespace.$ kubectl exec "$SLEEP_POD_FOO" -n foo -c sleep -- curl http://httpbin.bar:8000/html
upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
Cleanup
Remove the namespaces and uninstall Istio and cert-manager:
$ kubectl delete ns foo
$ kubectl delete ns bar
$ istioctl uninstall --purge -y
$ helm delete -n cert-manager cert-manager
$ kubectl delete ns istio-system cert-manager
$ unset ISTIOCA FOOCA BARCA
$ rm -rf istio.yaml proxyconfig-foo.yaml proxyconfig-bar.yaml selfsigned-issuer.yaml
Reasons to use this feature
Custom CA Integration - By specifying a Signer name in the Kubernetes CSR Request, this feature allows Istio to integrate with custom Certificate Authorities using the Kubernetes CSR API interface. This does require the custom CA to implement a Kubernetes controller to watch the
CertificateSigningRequest
Resources and act on them.Better multi-tenancy - By specifying a different cert-signer for different workloads, certificates for different tenant’s workloads can be signed by different CAs.