TLS troubleshooting

This page includes troubleshooting steps for configuring TLS certificates with the security plugin.

Validate YAML

opensearch.yml and the files in config/opensearch-security/ are in the YAML format. A linter like YAML Validator can help verify that you don’t have any formatting errors.

View contents of PEM certificates

You can use OpenSSL to display the content of each PEM certificate:

  1. openssl x509 -subject -nameopt RFC2253 -noout -in node1.pem

Then ensure that the value matches the one in opensearch.yml.

For more complete information on a certificate:

  1. openssl x509 -in node1.pem -text -noout

Check for special characters and whitespace in DNs

The security plugin uses the string representation of Distinguished Names (RFC1779) when validating node certificates.

If parts of your DN contain special characters (e.g. a comma), make sure you escape it in your configuration:

  1. plugins.security.nodes_dn:
  2.   - 'CN=node-0.example.com,OU=SSL,O=My\, Test,L=Test,C=DE'

You can have whitespace within a field, but not between fields.

Bad configuration

  1. plugins.security.nodes_dn:
  2.   - 'CN=node-0.example.com, OU=SSL,O=My\, Test, L=Test, C=DE'

Good configuration

  1. plugins.security.nodes_dn:
  2.   - 'CN=node-0.example.com,OU=SSL,O=My\, Test,L=Test,C=DE'

Check certificate IP addresses

Sometimes the IP address in your certificate is not the one communicating with the cluster. This problem can occur if your node has multiple interfaces or is running on a dual stack network (IPv6 and IPv4).

If this problem occurs, you might see the following in the node’s OpenSearch log:

  1. SSL Problem Received fatal alert: certificate_unknown javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

You might also see the following message in your cluster’s master log when the new node tries to join the cluster:

  1. Caused by: java.security.cert.CertificateException: No subject alternative names matching IP address 10.0.0.42 found

Check the IP address in the certificate:

  1. IPAddress: 2001:db8:0:1:1.2.3.4

In this example, the node tries to join the cluster with the IPv4 address of 10.0.0.42, but the certificate contians the IPv6 address of 2001:db8:0:1:1.2.3.4.

Validate certificate chain

TLS certificates are organized in a certificate chain. You can check with keytool that the certificate chain is correct by inspecting the owner and the issuer of each certificate. If you used the demo installation script that ships with the security plugin, the chain looks like:

Node certificate

  1. Owner: CN=node-0.example.com, OU=SSL, O=Test, L=Test, C=DE
  2. Issuer: CN=Example Com Inc. Signing CA, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com

Signing certificate

  1. Owner: CN=Example Com Inc. Signing CA, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com
  2. Issuer: CN=Example Com Inc. Root CA, OU=Example Com Inc. Root CA, O=Example Com Inc., DC=example, DC=com

Root certificate

  1. Owner: CN=Example Com Inc. Root CA, OU=Example Com Inc. Root CA, O=Example Com Inc., DC=example, DC=com
  2. Issuer: CN=Example Com Inc. Root CA, OU=Example Com Inc. Root CA, O=Example Com Inc., DC=example, DC=com

From the entries, you can see that the root certificate signed the intermediate certificate, which signed the node certificate. The root certificate signed itself, hence the name “self-signed certificate.” If you’re using separate keystore and truststore files, your root CA can most likely in the truststore.

Generally, the keystore contains client or node certificate and all intermediate certificates, and the truststore contains the root certificate.

Check the configured alias

If you have multiple entries in the keystore and you are using aliases to refer to them, make sure that the configured alias in opensearch.yml matches the one in the keystore. If there is only one entry in the keystore, you do not need to configure an alias.

View contents of your keystore and truststore

In order to view information about the certificates stored in your keystore or truststore, use the keytool command like:

  1. keytool -list -v -keystore keystore.jks

keytool prompts for the password of the keystore and lists all entries. For example, you can use this output to check for the correctness of the SAN and EKU settings.

Check SAN hostnames and IP addresses

The valid hostnames and IP addresses of a TLS certificates are stored as SAN entries. Check that the hostname and IP entries in the SAN section are correct, especially when you use hostname verification:

  1. Certificate[1]:
  2. Owner: CN=node-0.example.com, OU=SSL, O=Test, L=Test, C=DE
  3. ...
  4. Extensions:
  5. ...
  6. #5: ObjectId: 2.5.29.17 Criticality=false
  7. SubjectAlternativeName [
  8.   DNSName: node-0.example.com
  9.   DNSName: localhost
  10.   IPAddress: 127.0.0.1
  11.   ...
  12. ]

Check OID for node certificates

If you are using OIDs to denote valid node certificates, check that the SAN extension for your node certificate contains the correct OIDName:

  1. Certificate[1]:
  2. Owner: CN=node-0.example.com, OU=SSL, O=Test, L=Test, C=DE
  3. ...
  4. Extensions:
  5. ...
  6. #5: ObjectId: 2.5.29.17 Criticality=false
  7. SubjectAlternativeName [
  8.   ...
  9.   OIDName: 1.2.3.4.5.5
  10. ]

Check EKU field for node certificates

Node certificates need to have both serverAuth and clientAuth set in the extended key usage field:

  1. #3: ObjectId: 2.5.29.37 Criticality=false
  2. ExtendedKeyUsages [
  3.   serverAuth
  4.   clientAuth
  5. ]

TLS versions

The security plugin disables TLS version 1.0 by default; it is outdated, insecure, and vulnerable. If you need to use TLSv1 and accept the risks, you can enable it in opensearch.yml:

  1. plugins.security.ssl.http.enabled_protocols:
  2.   - "TLSv1"
  3.   - "TLSv1.1"
  4.   - "TLSv1.2"

Supported ciphers

TLS relies on the server and client negotiating a common cipher suite. Depending on your system, the available ciphers will vary. They depend on the JDK or OpenSSL version you’re using, and whether or not the JCE Unlimited Strength Jurisdiction Policy Files are installed.

For legal reasons, the JDK does not include strong ciphers like AES256. In order to use strong ciphers you need to download and install the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files. If you don’t have them installed, you might see an error message on startup:

  1. [INFO ] AES-256 not supported, max key length for AES is 128 bit.
  2. That is not an issue, it just limits possible encryption strength.
  3. To enable AES 256 install 'Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files'

The security plugin still works and falls back to weaker cipher suites. The plugin also prints out all available cipher suites during startup:

  1. [INFO ] sslTransportClientProvider:
  2. JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,
  3. TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, ...]