Transport Encryption for Knative Eventing

Flag name: transport-encryption

Stage: Alpha, disabled by default

Tracking issue: #5957

Overview

By default, event delivery within the cluster is unencrypted. This limits the types of events which can be transmitted to those of low compliance value (or a relaxed compliance posture) or, alternatively, forces administrators to use a service mesh or encrypted CNI to encrypt the traffic, which poses many challenges to Knative Eventing adopters.

Knative Brokers and Channels provides HTTPS endpoints to receive events. Given that these endpoints typically do not have public DNS names (e.g. svc.cluster.local or the like), these need to be signed by a non-public CA (cluster or organization specific CA).

Event producers are be able to connect to HTTPS endpoints with cluster-internal CA certificates.

Prerequisites

Installation

Eventing components use cert-manager issuers and certificates to provision TLS certificates and in the release assets, we release such default issuers and certificates that can be customized as necessary.

  1. Install issuers and certificates, run the following command:

    1. kubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.13.8/eventing-tls-networking.yaml
  2. Verify issuers and certificates are ready

    1. kubectl get certificates.cert-manager.io -n knative-eventing

    Example output:

    1. NAME READY SECRET AGE
    2. imc-dispatcher-server-tls True imc-dispatcher-server-tls 14s
    3. mt-broker-filter-server-tls True mt-broker-filter-server-tls 14s
    4. mt-broker-ingress-server-tls True mt-broker-ingress-server-tls 14s
    5. selfsigned-ca True eventing-ca 14s

Transport Encryption configuration

The transport-encryption feature flag is an enum configuration that configures how Addressables ( Broker, Channel, Sink) should accept events.

The possible values for transport-encryption are:

  • disabled (this is equivalent to the current behavior)
    • Addressables may accept events to HTTPS endpoints
    • Producers may send events to HTTPS endpoints
  • permissive
    • Addressables should accept events on both HTTP and HTTPS endpoints
    • Addressables should advertise both HTTP and HTTPS endpoints
    • Producers should prefer sending events to HTTPS endpoints, if available
  • strict
    • Addressables must not accept events to non-HTTPS endpoints
    • Addressables must only advertise HTTPS endpoints

For example, to enable strict transport encryption, the config-features ConfigMap will look like the following:

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4. name: config-features
  5. namespace: knative-eventing
  6. data:
  7. transport-encryption: "strict"

Trusting CA for a specific event sender

Event sources, triggers or subscriptions are considered event senders and they can be configured to trust specific CA certificates.

Important

The CA certs must be PEM formatted certificates. Since it’s a multi-line YAML string make sure that the CACerts value is indented correctly, otherwise when creating the resource it will be rejected.

Triggers and subscriptions can be configured as follow:

  1. spec:
  2. # ...
  3. subscriber:
  4. uri: https://mycorp-internal-example.com/v1/api
  5. CACerts: |-
  6. -----BEGIN CERTIFICATE-----
  7. MIIFWjCCA0KgAwIBAgIQT9Irj/VkyDOeTzRYZiNwYDANBgkqhkiG9w0BAQsFADBH
  8. MQswCQYDVQQGEwJDTjERMA8GA1UECgwIVW5pVHJ1c3QxJTAjBgNVBAMMHFVDQSBF
  9. eHRlbmRlZCBWYWxpZGF0aW9uIFJvb3QwHhcNMTUwMzEzMDAwMDAwWhcNMzgxMjMx
  10. MDAwMDAwWjBHMQswCQYDVQQGEwJDTjERMA8GA1UECgwIVW5pVHJ1c3QxJTAjBgNV
  11. BAMMHFVDQSBFeHRlbmRlZCBWYWxpZGF0aW9uIFJvb3QwggIiMA0GCSqGSIb3DQEB
  12. AQUAA4ICDwAwggIKAoICAQCpCQcoEwKwmeBkqh5DFnpzsZGgdT6o+uM4AHrsiWog
  13. D4vFsJszA1qGxliG1cGFu0/GnEBNyr7uaZa4rYEwmnySBesFK5pI0Lh2PpbIILvS
  14. sPGP2KxFRv+qZ2C0d35qHzwaUnoEPQc8hQ2E0B92CvdqFN9y4zR8V05WAT558aop
  15. O2z6+I9tTcg1367r3CTueUWnhbYFiN6IXSV8l2RnCdm/WhUFhvMJHuxYMjMR83dk
  16. sHYf5BA1FxvyDrFspCqjc/wJHx4yGVMR59mzLC52LqGj3n5qiAno8geK+LLNEOfi
  17. c0CTuwjRP+H8C5SzJe98ptfRr5//lpr1kXuYC3fUfugH0mK1lTnj8/FtDw5lhIpj
  18. VMWAtuCeS31HJqcBCF3RiJ7XwzJE+oJKCmhUfzhTA8ykADNkUVkLo4KRel7sFsLz
  19. KuZi2irbWWIQJUoqgQtHB0MGcIfS+pMRKXpITeuUx3BNr2fVUbGAIAEBtHoIppB/
  20. TuDvB0GHr2qlXov7z1CymlSvw4m6WC31MJixNnI5fkkE/SmnTHnkBVfblLkWU41G
  21. sx2VYVdWf6/wFlthWG82UBEL2KwrlRYaDh8IzTY0ZRBiZtWAXxQgXy0MoHgKaNYs
  22. 1+lvK9JKBZP8nm9rZ/+I8U6laUpSNwXqxhaN0sSZ0YIrO7o1dfdRUVjzyAfd5LQD
  23. fwIDAQABo0IwQDAdBgNVHQ4EFgQU2XQ65DA9DfcS3H5aBZ8eNJr34RQwDwYDVR0T
  24. AQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAYYwDQYJKoZIhvcNAQELBQADggIBADaN
  25. l8xCFWQpN5smLNb7rhVpLGsaGvdftvkHTFnq88nIua7Mui563MD1sC3AO6+fcAUR
  26. ap8lTwEpcOPlDOHqWnzcSbvBHiqB9RZLcpHIojG5qtr8nR/zXUACE/xOHAbKsxSQ
  27. VBcZEhrxH9cMaVr2cXj0lH2RC47skFSOvG+hTKv8dGT9cZr4QQehzZHkPJrgmzI5
  28. c6sq1WnIeJEmMX3ixzDx/BR4dxIOE/TdFpS/S2d7cFOFyrC78zhNLJA5wA3CXWvp
  29. 4uXViI3WLL+rG761KIcSF3Ru/H38j9CHJrAb+7lsq+KePRXBOy5nAliRn+/4Qh8s
  30. t2j1da3Ptfb/EX3C8CSlrdP6oDyp+l3cpaDvRKS+1ujl5BOWF3sGPjLtx7dCvHaj
  31. 2GU4Kzg1USEODm8uNBNA4StnDG1KQTAYI1oyVZnJF+A83vbsea0rWBmirSwiGpWO
  32. vpaQXUJXxPkUAzUrHC1RVwinOt4/5Mi0A3PCwSaAuwtCH60NryZy2sy+s6ODWA2C
  33. xR9GUeOcGMyNm43sSet1UNWMKFnKdDTajAshqx7qG+XH/RU+wBeq+yNuJkbL+vmx
  34. cmtpzyKEC2IPrNkZAJSidjzULZrtBJ4tBmIQN1IchXIbJ+XMxjHsN+xjWZsLHXbM
  35. fjKaiJUINlK73nZfdklJrX+9ZSCyycErdhh2n1ax
  36. -----END CERTIFICATE-----

Similarly, sources can be configured as follow:

  1. spec:
  2. # ...
  3. sink:
  4. uri: https://mycorp-internal-example.com/v1/api
  5. CACerts: |-
  6. -----BEGIN CERTIFICATE-----
  7. MIIFWjCCA0KgAwIBAgIQT9Irj/VkyDOeTzRYZiNwYDANBgkqhkiG9w0BAQsFADBH
  8. MQswCQYDVQQGEwJDTjERMA8GA1UECgwIVW5pVHJ1c3QxJTAjBgNVBAMMHFVDQSBF
  9. eHRlbmRlZCBWYWxpZGF0aW9uIFJvb3QwHhcNMTUwMzEzMDAwMDAwWhcNMzgxMjMx
  10. MDAwMDAwWjBHMQswCQYDVQQGEwJDTjERMA8GA1UECgwIVW5pVHJ1c3QxJTAjBgNV
  11. BAMMHFVDQSBFeHRlbmRlZCBWYWxpZGF0aW9uIFJvb3QwggIiMA0GCSqGSIb3DQEB
  12. AQUAA4ICDwAwggIKAoICAQCpCQcoEwKwmeBkqh5DFnpzsZGgdT6o+uM4AHrsiWog
  13. D4vFsJszA1qGxliG1cGFu0/GnEBNyr7uaZa4rYEwmnySBesFK5pI0Lh2PpbIILvS
  14. sPGP2KxFRv+qZ2C0d35qHzwaUnoEPQc8hQ2E0B92CvdqFN9y4zR8V05WAT558aop
  15. O2z6+I9tTcg1367r3CTueUWnhbYFiN6IXSV8l2RnCdm/WhUFhvMJHuxYMjMR83dk
  16. sHYf5BA1FxvyDrFspCqjc/wJHx4yGVMR59mzLC52LqGj3n5qiAno8geK+LLNEOfi
  17. c0CTuwjRP+H8C5SzJe98ptfRr5//lpr1kXuYC3fUfugH0mK1lTnj8/FtDw5lhIpj
  18. VMWAtuCeS31HJqcBCF3RiJ7XwzJE+oJKCmhUfzhTA8ykADNkUVkLo4KRel7sFsLz
  19. KuZi2irbWWIQJUoqgQtHB0MGcIfS+pMRKXpITeuUx3BNr2fVUbGAIAEBtHoIppB/
  20. TuDvB0GHr2qlXov7z1CymlSvw4m6WC31MJixNnI5fkkE/SmnTHnkBVfblLkWU41G
  21. sx2VYVdWf6/wFlthWG82UBEL2KwrlRYaDh8IzTY0ZRBiZtWAXxQgXy0MoHgKaNYs
  22. 1+lvK9JKBZP8nm9rZ/+I8U6laUpSNwXqxhaN0sSZ0YIrO7o1dfdRUVjzyAfd5LQD
  23. fwIDAQABo0IwQDAdBgNVHQ4EFgQU2XQ65DA9DfcS3H5aBZ8eNJr34RQwDwYDVR0T
  24. AQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAYYwDQYJKoZIhvcNAQELBQADggIBADaN
  25. l8xCFWQpN5smLNb7rhVpLGsaGvdftvkHTFnq88nIua7Mui563MD1sC3AO6+fcAUR
  26. ap8lTwEpcOPlDOHqWnzcSbvBHiqB9RZLcpHIojG5qtr8nR/zXUACE/xOHAbKsxSQ
  27. VBcZEhrxH9cMaVr2cXj0lH2RC47skFSOvG+hTKv8dGT9cZr4QQehzZHkPJrgmzI5
  28. c6sq1WnIeJEmMX3ixzDx/BR4dxIOE/TdFpS/S2d7cFOFyrC78zhNLJA5wA3CXWvp
  29. 4uXViI3WLL+rG761KIcSF3Ru/H38j9CHJrAb+7lsq+KePRXBOy5nAliRn+/4Qh8s
  30. t2j1da3Ptfb/EX3C8CSlrdP6oDyp+l3cpaDvRKS+1ujl5BOWF3sGPjLtx7dCvHaj
  31. 2GU4Kzg1USEODm8uNBNA4StnDG1KQTAYI1oyVZnJF+A83vbsea0rWBmirSwiGpWO
  32. vpaQXUJXxPkUAzUrHC1RVwinOt4/5Mi0A3PCwSaAuwtCH60NryZy2sy+s6ODWA2C
  33. xR9GUeOcGMyNm43sSet1UNWMKFnKdDTajAshqx7qG+XH/RU+wBeq+yNuJkbL+vmx
  34. cmtpzyKEC2IPrNkZAJSidjzULZrtBJ4tBmIQN1IchXIbJ+XMxjHsN+xjWZsLHXbM
  35. fjKaiJUINlK73nZfdklJrX+9ZSCyycErdhh2n1ax
  36. -----END CERTIFICATE-----

Verifying that the feature is working

Save the following YAML into a file called default-broker-example.yaml

  1. # default-broker-example.yaml
  2. apiVersion: eventing.knative.dev/v1
  3. kind: Broker
  4. metadata:
  5. name: br
  6. ---
  7. apiVersion: eventing.knative.dev/v1
  8. kind: Trigger
  9. metadata:
  10. name: tr
  11. spec:
  12. broker: br
  13. subscriber:
  14. ref:
  15. apiVersion: v1
  16. kind: Service
  17. name: event-display
  18. ---
  19. apiVersion: v1
  20. kind: Service
  21. metadata:
  22. name: event-display
  23. spec:
  24. selector:
  25. app: event-display
  26. ports:
  27. - protocol: TCP
  28. port: 80
  29. targetPort: 8080
  30. ---
  31. apiVersion: v1
  32. kind: Pod
  33. metadata:
  34. name: event-display
  35. labels:
  36. app: event-display
  37. spec:
  38. containers:
  39. - name: event-display
  40. image: gcr.io/knative-releases/knative.dev/eventing/cmd/event_display
  41. imagePullPolicy: Always
  42. ports:
  43. - containerPort: 8080

Apply the default-broker-example.yaml file into a test namespace transport-encryption-test:

  1. kubectl create namespace transport-encryption-test
  2. kubectl apply -n transport-encryption-test -f defautl-broker-example.yaml

Verify that addresses are all HTTPS:

  1. kubectl get brokers.eventing.knative.dev -n transport-encryption-test br -oyaml

Example output:

  1. apiVersion: eventing.knative.dev/v1
  2. kind: Broker
  3. metadata:
  4. # ...
  5. name: br
  6. namespace: transport-encryption-test
  7. # ...
  8. status:
  9. address:
  10. CACerts: |
  11. -----BEGIN CERTIFICATE-----
  12. MIIBbzCCARagAwIBAgIQAur7vdEcreEWSEQatCYlNjAKBggqhkjOPQQDAjAYMRYw
  13. FAYDVQQDEw1zZWxmc2lnbmVkLWNhMB4XDTIzMDgwMzA4MzA1N1oXDTIzMTEwMTA4
  14. MzA1N1owGDEWMBQGA1UEAxMNc2VsZnNpZ25lZC1jYTBZMBMGByqGSM49AgEGCCqG
  15. SM49AwEHA0IABBqkD9lAwrnXCo/OOdpKzJROSbzCeC73FE/Np+/j8n862Ox5xYwJ
  16. tAp/o3RDpDa3omhzqZoYumqdtneozGFY/zGjQjBAMA4GA1UdDwEB/wQEAwICpDAP
  17. BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSHoKjXzfxfudt3mVGU3VudSi6TrTAK
  18. BggqhkjOPQQDAgNHADBEAiA5z0/TpD7T6vRpN9VQisQMtum/Zg3tThnYA5nFnAW7
  19. KAIgKR/EzW7f8BPcnlcgXt5kp3Fdqye1SAkjxZzr2a0Zik8=
  20. -----END CERTIFICATE-----
  21. name: https
  22. url: https://broker-ingress.knative-eventing.svc.cluster.local/transport-encryption-test/br
  23. addresses:
  24. - CACerts: |
  25. -----BEGIN CERTIFICATE-----
  26. MIIBbzCCARagAwIBAgIQAur7vdEcreEWSEQatCYlNjAKBggqhkjOPQQDAjAYMRYw
  27. FAYDVQQDEw1zZWxmc2lnbmVkLWNhMB4XDTIzMDgwMzA4MzA1N1oXDTIzMTEwMTA4
  28. MzA1N1owGDEWMBQGA1UEAxMNc2VsZnNpZ25lZC1jYTBZMBMGByqGSM49AgEGCCqG
  29. SM49AwEHA0IABBqkD9lAwrnXCo/OOdpKzJROSbzCeC73FE/Np+/j8n862Ox5xYwJ
  30. tAp/o3RDpDa3omhzqZoYumqdtneozGFY/zGjQjBAMA4GA1UdDwEB/wQEAwICpDAP
  31. BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSHoKjXzfxfudt3mVGU3VudSi6TrTAK
  32. BggqhkjOPQQDAgNHADBEAiA5z0/TpD7T6vRpN9VQisQMtum/Zg3tThnYA5nFnAW7
  33. KAIgKR/EzW7f8BPcnlcgXt5kp3Fdqye1SAkjxZzr2a0Zik8=
  34. -----END CERTIFICATE-----
  35. name: https
  36. url: https://broker-ingress.knative-eventing.svc.cluster.local/transport-encryption-test/br
  37. annotations:
  38. knative.dev/channelAPIVersion: messaging.knative.dev/v1
  39. knative.dev/channelAddress: https://imc-dispatcher.knative-eventing.svc.cluster.local/transport-encryption-test/br-kne-trigger
  40. knative.dev/channelCACerts: |
  41. -----BEGIN CERTIFICATE-----
  42. MIIBbzCCARagAwIBAgIQAur7vdEcreEWSEQatCYlNjAKBggqhkjOPQQDAjAYMRYw
  43. FAYDVQQDEw1zZWxmc2lnbmVkLWNhMB4XDTIzMDgwMzA4MzA1N1oXDTIzMTEwMTA4
  44. MzA1N1owGDEWMBQGA1UEAxMNc2VsZnNpZ25lZC1jYTBZMBMGByqGSM49AgEGCCqG
  45. SM49AwEHA0IABBqkD9lAwrnXCo/OOdpKzJROSbzCeC73FE/Np+/j8n862Ox5xYwJ
  46. tAp/o3RDpDa3omhzqZoYumqdtneozGFY/zGjQjBAMA4GA1UdDwEB/wQEAwICpDAP
  47. BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSHoKjXzfxfudt3mVGU3VudSi6TrTAK
  48. BggqhkjOPQQDAgNHADBEAiA5z0/TpD7T6vRpN9VQisQMtum/Zg3tThnYA5nFnAW7
  49. KAIgKR/EzW7f8BPcnlcgXt5kp3Fdqye1SAkjxZzr2a0Zik8=
  50. -----END CERTIFICATE-----
  51. knative.dev/channelKind: InMemoryChannel
  52. knative.dev/channelName: br-kne-trigger
  53. conditions:
  54. # ...

Sending events to the Broker using HTTPS endpoints:

  1. kubectl run curl -n transport-encryption-test --image=curlimages/curl -i --tty -- sh

Save the CA certs from the Broker’s .status.address.CACerts field into /tmp/cacerts.pem

  1. cat <<EOF >> /tmp/cacerts.pem
  2. -----BEGIN CERTIFICATE-----
  3. MIIBbzCCARagAwIBAgIQAur7vdEcreEWSEQatCYlNjAKBggqhkjOPQQDAjAYMRYw
  4. FAYDVQQDEw1zZWxmc2lnbmVkLWNhMB4XDTIzMDgwMzA4MzA1N1oXDTIzMTEwMTA4
  5. MzA1N1owGDEWMBQGA1UEAxMNc2VsZnNpZ25lZC1jYTBZMBMGByqGSM49AgEGCCqG
  6. SM49AwEHA0IABBqkD9lAwrnXCo/OOdpKzJROSbzCeC73FE/Np+/j8n862Ox5xYwJ
  7. tAp/o3RDpDa3omhzqZoYumqdtneozGFY/zGjQjBAMA4GA1UdDwEB/wQEAwICpDAP
  8. BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSHoKjXzfxfudt3mVGU3VudSi6TrTAK
  9. BggqhkjOPQQDAgNHADBEAiA5z0/TpD7T6vRpN9VQisQMtum/Zg3tThnYA5nFnAW7
  10. KAIgKR/EzW7f8BPcnlcgXt5kp3Fdqye1SAkjxZzr2a0Zik8=
  11. -----END CERTIFICATE-----
  12. EOF

Send the event by running the following command:

  1. curl -v -X POST -H "content-type: application/json" -H "ce-specversion: 1.0" -H "ce-source: my/curl/command" -H "ce-type: my.demo.event" -H "ce-id: 6cf17c7b-30b1-45a6-80b0-4cf58c92b947" -d '{"name":"Knative Demo"}' --cacert /tmp/cacert
  2. s.pem https://broker-ingress.knative-eventing.svc.cluster.local/transport-encryption-test/br

Example output:

  1. * processing: https://broker-ingress.knative-eventing.svc.cluster.local/transport-encryption-test/br
  2. * Trying 10.96.174.249:443...
  3. * Connected to broker-ingress.knative-eventing.svc.cluster.local (10.96.174.249) port 443
  4. * ALPN: offers h2,http/1.1
  5. * TLSv1.3 (OUT), TLS handshake, Client hello (1):
  6. * CAfile: /tmp/cacerts.pem
  7. * CApath: none
  8. * TLSv1.3 (IN), TLS handshake, Server hello (2):
  9. * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  10. * TLSv1.3 (IN), TLS handshake, Certificate (11):
  11. * TLSv1.3 (IN), TLS handshake, CERT verify (15):
  12. * TLSv1.3 (IN), TLS handshake, Finished (20):
  13. * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  14. * TLSv1.3 (OUT), TLS handshake, Finished (20):
  15. * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
  16. * ALPN: server accepted h2
  17. * Server certificate:
  18. * subject: O=local
  19. * start date: Aug 3 08:31:02 2023 GMT
  20. * expire date: Nov 1 08:31:02 2023 GMT
  21. * subjectAltName: host "broker-ingress.knative-eventing.svc.cluster.local" matched cert's "broker-ingress.knative-eventing.svc.cluster.local"
  22. * issuer: CN=selfsigned-ca
  23. * SSL certificate verify ok.
  24. * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  25. * using HTTP/2
  26. * h2 [:method: POST]
  27. * h2 [:scheme: https]
  28. * h2 [:authority: broker-ingress.knative-eventing.svc.cluster.local]
  29. * h2 [:path: /transport-encryption-test/br]
  30. * h2 [user-agent: curl/8.2.1]
  31. * h2 [accept: */*]
  32. * h2 [content-type: application/json]
  33. * h2 [ce-specversion: 1.0]
  34. * h2 [ce-source: my/curl/command]
  35. * h2 [ce-type: my.demo.event]
  36. * h2 [ce-id: 6cf17c7b-30b1-45a6-80b0-4cf58c92b947]
  37. * h2 [content-length: 23]
  38. * Using Stream ID: 1
  39. > POST /transport-encryption-test/br HTTP/2
  40. > Host: broker-ingress.knative-eventing.svc.cluster.local
  41. > User-Agent: curl/8.2.1
  42. > Accept: */*
  43. > content-type: application/json
  44. > ce-specversion: 1.0
  45. > ce-source: my/curl/command
  46. > ce-type: my.demo.event
  47. > ce-id: 6cf17c7b-30b1-45a6-80b0-4cf58c92b947
  48. > Content-Length: 23
  49. >
  50. < HTTP/2 202
  51. < allow: POST, OPTIONS
  52. < content-length: 0
  53. < date: Thu, 03 Aug 2023 10:08:22 GMT
  54. <
  55. * Connection #0 to host broker-ingress.knative-eventing.svc.cluster.local left intact