Using TLS with KeyStore configure

Overview

Apache Pulsar supports TLS encryption and TLS authentication between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use KeyStore type configure for TLS.

TLS encryption with KeyStore configure

Generate TLS key and certificate

The first step of deploying TLS is to generate the key and the certificate for each machine in the cluster. You can use Java’s keytool utility to accomplish this task. We will generate the key into a temporary keystore initially for broker, so that we can export and sign it later with CA.

  2. keytool -keystore broker.keystore.jks -alias localhost -validity {validity} -genkeypair -keyalg RSA

You need to specify two parameters in the above command:

  1. keystore: the keystore file that stores the certificate. The keystore file contains the private key of the certificate; hence, it needs to be kept safely.
  2. validity: the valid time of the certificate in days.

Ensure that common name (CN) matches exactly with the fully qualified domain name (FQDN) of the server. The client compares the CN with the DNS domain name to ensure that it is indeed connecting to the desired server, not a malicious one.

Creating your own CA

After the first step, each broker in the cluster has a public-private key pair, and a certificate to identify the machine. The certificate, however, is unsigned, which means that an attacker can create such a certificate to pretend to be any machine.

Therefore, it is important to prevent forged certificates by signing them for each machine in the cluster. A certificate authority (CA) is responsible for signing certificates. CA works likes a government that issues passports — the government stamps (signs) each passport so that the passport becomes difficult to forge. Other governments verify the stamps to ensure the passport is authentic. Similarly, the CA signs the certificates, and the cryptography guarantees that a signed certificate is computationally difficult to forge. Thus, as long as the CA is a genuine and trusted authority, the clients have high assurance that they are connecting to the authentic machines.

  2. openssl req -new -x509 -keyout ca-key -out ca-cert -days 365

The generated CA is simply a public-private key pair and certificate, and it is intended to sign other certificates.

The next step is to add the generated CA to the clients’ truststore so that the clients can trust this CA:

  2. keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert

NOTE: If you configure the brokers to require client authentication by setting tlsRequireTrustedClientCertOnConnect to true on the broker configuration, then you must also provide a truststore for the brokers and it should have all the CA certificates that clients keys were signed by.

  2. keytool -keystore broker.truststore.jks -alias CARoot -import -file ca-cert

In contrast to the keystore, which stores each machine’s own identity, the truststore of a client stores all the certificates that the client should trust. Importing a certificate into one’s truststore also means trusting all certificates that are signed by that certificate. As the analogy above, trusting the government (CA) also means trusting all passports (certificates) that it has issued. This attribute is called the chain of trust, and it is particularly useful when deploying TLS on a large BookKeeper cluster. You can sign all certificates in the cluster with a single CA, and have all machines share the same truststore that trusts the CA. That way all machines can authenticate all other machines.

Signing the certificate

The next step is to sign all certificates in the keystore with the CA we generated. First, you need to export the certificate from the keystore:

  2. keytool -keystore broker.keystore.jks -alias localhost -certreq -file cert-file

Then sign it with the CA:

  2. openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days {validity} -CAcreateserial -passin pass:{ca-password}

Finally, you need to import both the certificate of the CA and the signed certificate into the keystore:

  2. keytool -keystore broker.keystore.jks -alias CARoot -import -file ca-cert
  3. keytool -keystore broker.keystore.jks -alias localhost -import -file cert-signed

The definitions of the parameters are the following:

  1. keystore: the location of the keystore
  2. ca-cert: the certificate of the CA
  3. ca-key: the private key of the CA
  4. ca-password: the passphrase of the CA
  5. cert-file: the exported, unsigned certificate of the broker
  6. cert-signed: the signed certificate of the broker

Configuring brokers

Brokers enable TLS by provide valid brokerServicePortTls and webServicePortTls, and also need set tlsEnabledWithKeyStore to true for using KeyStore type configuration. Besides this, KeyStore path, KeyStore password, TrustStore path, and TrustStore password need to provided. And since broker will create internal client/admin client to communicate with other brokers, user also need to provide config for them, this is similar to how user config the outside client/admin-client. If tlsRequireTrustedClientCertOnConnect is true, broker will reject the Connection if the Client Certificate is not trusted.

The following TLS configs are needed on the broker side:

  2. tlsEnabledWithKeyStore=true
  3. # key store
  4. tlsKeyStoreType=JKS
  5. tlsKeyStore=/var/private/tls/broker.keystore.jks
  6. tlsKeyStorePassword=brokerpw
  8. # trust store
  9. tlsTrustStoreType=JKS
  10. tlsTrustStore=/var/private/tls/broker.truststore.jks
  11. tlsTrustStorePassword=brokerpw
  13. # internal client/admin-client config
  14. brokerClientTlsEnabled=true
  15. brokerClientTlsEnabledWithKeyStore=true
  16. brokerClientTlsTrustStoreType=JKS
  17. brokerClientTlsTrustStore=/var/private/tls/client.truststore.jks
  18. brokerClientTlsTrustStorePassword=clientpw

NOTE: it is important to restrict access to the store files via filesystem permissions.

If you have configured TLS on the broker, to disable non-TLS ports, you can set the values of the following configurations to empty as below.

  2. brokerServicePort=
  3. webServicePort=

In this case, you need to set the following configurations.

  2. brokerClientTlsEnabled=true // Set this to true
  3. brokerClientTlsEnabledWithKeyStore=true  // Set this to true
  4. brokerClientTlsTrustStore= // Set this to your desired value
  5. brokerClientTlsTrustStorePassword= // Set this to your desired value

Optional settings that may worth consider:

  1. tlsClientAuthentication=false: Enable/Disable using TLS for authentication. This config when enabled will authenticate the other end of the communication channel. It should be enabled on both brokers and clients for mutual TLS.
  2. tlsCiphers=[TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256], A cipher suite is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS network protocol. By default, it is null. OpenSSL CiphersJDK Ciphers
  3. tlsProtocols=[TLSv1.3,TLSv1.2] (list out the TLS protocols that you are going to accept from clients). By default, it is not set.

Configuring Clients

This is similar to TLS encryption configuing for client with PEM type. For a minimal configuration, you need to provide the TrustStore information.

For example:

  1. for Command-line tools like pulsar-admin, pulsar-perf, and pulsar-client use the conf/client.conf config file in a Pulsar installation.

    2. webServiceUrl=https://broker.example.com:8443/
    3. brokerServiceUrl=pulsar+ssl://broker.example.com:6651/
    4. useKeyStoreTls=true
    5. tlsTrustStoreType=JKS
    6. tlsTrustStorePath=/var/private/tls/client.truststore.jks
    7. tlsTrustStorePassword=clientpw

  1. for java client

    2. import org.apache.pulsar.client.api.PulsarClient;
    4. PulsarClient client = PulsarClient.builder()
    5.     .serviceUrl("pulsar+ssl://broker.example.com:6651/")
    6.     .enableTls(true)
    7.     .useKeyStoreTls(true)
    8.     .tlsTrustStorePath("/var/private/tls/client.truststore.jks")
    9.     .tlsTrustStorePassword("clientpw")
    10.     .allowTlsInsecureConnection(false)
    11.     .build();

  1. for java admin client

    2.     PulsarAdmin amdin = PulsarAdmin.builder().serviceHttpUrl("https://broker.example.com:8443")
    3.         .useKeyStoreTls(true)
    4.         .tlsTrustStorePath("/var/private/tls/client.truststore.jks")
    5.         .tlsTrustStorePassword("clientpw")
    6.         .allowTlsInsecureConnection(false)
    7.         .build();

Note: Please configure tlsTrustStorePath when you set useKeyStoreTls to true.

TLS authentication with KeyStore configure

This similar to TLS authentication with PEM type

broker authentication config

broker.conf

  2. # Configuration to enable authentication
  3. authenticationEnabled=true
  4. authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls
  6. # this should be the CN for one of client keystore.
  7. superUserRoles=admin
  9. # Enable KeyStore type
  10. tlsEnabledWithKeyStore=true
  11. requireTrustedClientCertOnConnect=true
  13. # key store
  14. tlsKeyStoreType=JKS
  15. tlsKeyStore=/var/private/tls/broker.keystore.jks
  16. tlsKeyStorePassword=brokerpw
  18. # trust store
  19. tlsTrustStoreType=JKS
  20. tlsTrustStore=/var/private/tls/broker.truststore.jks
  21. tlsTrustStorePassword=brokerpw
  23. # internal client/admin-client config
  24. brokerClientTlsEnabled=true
  25. brokerClientTlsEnabledWithKeyStore=true
  26. brokerClientTlsTrustStoreType=JKS
  27. brokerClientTlsTrustStore=/var/private/tls/client.truststore.jks
  28. brokerClientTlsTrustStorePassword=clientpw
  29. # internal auth config
  30. brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationKeyStoreTls
  31. brokerClientAuthenticationParameters={"keyStoreType":"JKS","keyStorePath":"/var/private/tls/client.keystore.jks","keyStorePassword":"clientpw"}
  32. # currently websocket not support keystore type
  33. webSocketServiceEnabled=false

client authentication configuring

Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client.

For example:

  1. for Command-line tools like pulsar-admin, pulsar-perf, and pulsar-client use the conf/client.conf config file in a Pulsar installation.

    2. webServiceUrl=https://broker.example.com:8443/
    3. brokerServiceUrl=pulsar+ssl://broker.example.com:6651/
    4. useKeyStoreTls=true
    5. tlsTrustStoreType=JKS
    6. tlsTrustStorePath=/var/private/tls/client.truststore.jks
    7. tlsTrustStorePassword=clientpw
    8. authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationKeyStoreTls
    9. authParams={"keyStoreType":"JKS","keyStorePath":"/path/to/keystorefile","keyStorePassword":"keystorepw"}

  1. for java client

    2. import org.apache.pulsar.client.api.PulsarClient;
    4. PulsarClient client = PulsarClient.builder()
    5.     .serviceUrl("pulsar+ssl://broker.example.com:6651/")
    6.     .enableTls(true)
    7.     .useKeyStoreTls(true)
    8.     .tlsTrustStorePath("/var/private/tls/client.truststore.jks")
    9.     .tlsTrustStorePassword("clientpw")
    10.     .allowTlsInsecureConnection(false)
    11.     .authentication(
    12.             "org.apache.pulsar.client.impl.auth.AuthenticationKeyStoreTls",
    13.             "keyStoreType:JKS,keyStorePath:/var/private/tls/client.keystore.jks,keyStorePassword:clientpw")
    14.     .build();

  1. for java admin client

    2.     PulsarAdmin amdin = PulsarAdmin.builder().serviceHttpUrl("https://broker.example.com:8443")
    3.         .useKeyStoreTls(true)
    4.         .tlsTrustStorePath("/var/private/tls/client.truststore.jks")
    5.         .tlsTrustStorePassword("clientpw")
    6.         .allowTlsInsecureConnection(false)
    7.         .authentication(
    8.                "org.apache.pulsar.client.impl.auth.AuthenticationKeyStoreTls",
    9.                "keyStoreType:JKS,keyStorePath:/var/private/tls/client.keystore.jks,keyStorePassword:clientpw")
    10.         .build();

Note: Please configure tlsTrustStorePath when you set useKeyStoreTls to true.

Enabling TLS Logging

You can enable TLS debug logging at the JVM level by starting the brokers and/or clients with javax.net.debug system property. For example:

  2. -Djavax.net.debug=all

You can find more details on this in Oracle documentation on debugging SSL/TLS connections.