1 Using certificates

Overview

Zabbix can use RSA certificates in PEM format, signed by a public or in-house certificate authority (CA). Certificate verification is done against a pre-configured CA certificate. Optionally certificate revocation lists (CRL) can be used. Each Zabbix component can have only one certificate configured.

For more information how to set up and operate internal CA, how to generate certificate requests and sign them, how to revoke certificates you can find numerous online how-tos, for example, OpenSSL PKI Tutorial v1.1 .

Carefully consider and test your certificate extensions - see Limitations on using X.509 v3 certificate extensions.

Certificate configuration parameters

ParameterMandatoryDescription
TLSCAFileyesFull pathname of a file containing the top-level CA(s) certificates for peer certificate verification.
In case of certificate chain with several members they must be ordered: lower level CA certificates first followed by certificates of higher level CA(s).
Certificates from multiple CA(s) can be included in a single file.
TLSCRLFilenoFull pathname of a file containing Certificate Revocation Lists. See notes in Certificate Revocation Lists (CRL).
TLSCertFileyesFull pathname of a file containing certificate (certificate chain).
In case of certificate chain with several members they must be ordered: server, proxy, or agent certificate first, followed by lower level CA certificates then certificates of higher level CA(s).
TLSKeyFileyesFull pathname of a file containing private key. Set access rights to this file - it must be readable only by Zabbix user.
TLSServerCertIssuernoAllowed server certificate issuer.
TLSServerCertSubjectnoAllowed server certificate subject.

Configuring certificate on Zabbix server

1. In order to verify peer certificates, Zabbix server must have access to file with their top-level self-signed root CA certificates. For example, if we expect certificates from two independent root CAs, we can put their certificates into file /home/zabbix/zabbix_ca_file like this:

  1. Certificate:
  2.            Data:
  3.                Version: 3 (0x2)
  4.                Serial Number: 1 (0x1)
  5.            Signature Algorithm: sha1WithRSAEncryption
  6.                Issuer: DC=com, DC=zabbix, O=Zabbix SIA, OU=Development group, CN=Root1 CA
  7.                    ...
  8.                Subject: DC=com, DC=zabbix, O=Zabbix SIA, OU=Development group, CN=Root1 CA
  9.                Subject Public Key Info:
  10.                    Public Key Algorithm: rsaEncryption
  11.                        Public-Key: (2048 bit)
  12.                    ...
  13.                X509v3 extensions:
  14.                    X509v3 Key Usage: critical
  15.                        Certificate Sign, CRL Sign
  16.                    X509v3 Basic Constraints: critical
  17.                        CA:TRUE
  18.                    ...
  19.        -----BEGIN CERTIFICATE-----
  20.        MIID2jCCAsKgAwIBAgIBATANBgkqhkiG9w0BAQUFADB+MRMwEQYKCZImiZPyLGQB
  21.        ....
  22.        9wEzdN8uTrqoyU78gi12npLj08LegRKjb5hFTVmO
  23.        -----END CERTIFICATE-----
  24.        Certificate:
  25.            Data:
  26.                Version: 3 (0x2)
  27.                Serial Number: 1 (0x1)
  28.            Signature Algorithm: sha1WithRSAEncryption
  29.                Issuer: DC=com, DC=zabbix, O=Zabbix SIA, OU=Development group, CN=Root2 CA
  30.                    ...
  31.                Subject: DC=com, DC=zabbix, O=Zabbix SIA, OU=Development group, CN=Root2 CA
  32.                Subject Public Key Info:
  33.                    Public Key Algorithm: rsaEncryption
  34.                        Public-Key: (2048 bit)
  35.                    ....
  36.                X509v3 extensions:
  37.                    X509v3 Key Usage: critical
  38.                        Certificate Sign, CRL Sign
  39.                    X509v3 Basic Constraints: critical
  40.                        CA:TRUE
  41.                    ....       
  42.        -----BEGIN CERTIFICATE-----
  43.        MIID3DCCAsSgAwIBAgIBATANBgkqhkiG9w0BAQUFADB/MRMwEQYKCZImiZPyLGQB
  44.        ...
  45.        vdGNYoSfvu41GQAR5Vj5FnRJRzv5XQOZ3B6894GY1zY=
  46.        -----END CERTIFICATE-----

2. Put Zabbix server certificate chain into file, for example, /home/zabbix/zabbix_server.crt:

  1. Certificate:
  2.            Data:
  3.                Version: 3 (0x2)
  4.                Serial Number: 1 (0x1)
  5.            Signature Algorithm: sha1WithRSAEncryption
  6.                Issuer: DC=com, DC=zabbix, O=Zabbix SIA, OU=Development group, CN=Signing CA
  7.                ...
  8.                Subject: DC=com, DC=zabbix, O=Zabbix SIA, OU=Development group, CN=Zabbix server
  9.                Subject Public Key Info:
  10.                    Public Key Algorithm: rsaEncryption
  11.                        Public-Key: (2048 bit)
  12.                        ...
  13.                X509v3 extensions:
  14.                    X509v3 Key Usage: critical
  15.                        Digital Signature, Key Encipherment
  16.                    X509v3 Basic Constraints: 
  17.                        CA:FALSE
  18.                    ...
  19.        -----BEGIN CERTIFICATE-----
  20.        MIIECDCCAvCgAwIBAgIBATANBgkqhkiG9w0BAQUFADCBgTETMBEGCgmSJomT8ixk
  21.        ...
  22.        h02u1GHiy46GI+xfR3LsPwFKlkTaaLaL/6aaoQ==
  23.        -----END CERTIFICATE-----
  24.        Certificate:
  25.            Data:
  26.                Version: 3 (0x2)
  27.                Serial Number: 2 (0x2)
  28.            Signature Algorithm: sha1WithRSAEncryption
  29.                Issuer: DC=com, DC=zabbix, O=Zabbix SIA, OU=Development group, CN=Root1 CA
  30.                ...
  31.                Subject: DC=com, DC=zabbix, O=Zabbix SIA, OU=Development group, CN=Signing CA
  32.                Subject Public Key Info:
  33.                    Public Key Algorithm: rsaEncryption
  34.                        Public-Key: (2048 bit)
  35.                    ...
  36.                X509v3 extensions:
  37.                    X509v3 Key Usage: critical
  38.                        Certificate Sign, CRL Sign
  39.                    X509v3 Basic Constraints: critical
  40.                        CA:TRUE, pathlen:0
  41.                ...
  42.        -----BEGIN CERTIFICATE-----
  43.        MIID4TCCAsmgAwIBAgIBAjANBgkqhkiG9w0BAQUFADB+MRMwEQYKCZImiZPyLGQB
  44.        ...
  45.        dyCeWnvL7u5sd6ffo8iRny0QzbHKmQt/wUtcVIvWXdMIFJM0Hw==
  46.        -----END CERTIFICATE-----

Here the first is Zabbix server certificate, followed by intermediate CA certificate.

Use of any attributes except of the ones mentioned above is discouraged for both client and server certificates, because it may affect certificate verification process. For example, OpenSSL might fail to establish encrypted connection if X509v3 Extended Key Usage or Netscape Cert Type are set. See also: Limitations on using X.509 v3 certificate extensions.

3. Put Zabbix server private key into file, for example, /home/zabbix/zabbix_server.key:

  1. -----BEGIN PRIVATE KEY-----
  2.        MIIEwAIBADANBgkqhkiG9w0BAQEFAASCBKowggSmAgEAAoIBAQC9tIXIJoVnNXDl
  3.        ...
  4.        IJLkhbybBYEf47MLhffWa7XvZTY=
  5.        -----END PRIVATE KEY-----

4. Edit TLS parameters in Zabbix server configuration file like this:

  1. TLSCAFile=/home/zabbix/zabbix_ca_file
  2.        TLSCertFile=/home/zabbix/zabbix_server.crt
  3.        TLSKeyFile=/home/zabbix/zabbix_server.key

Configuring certificate-based encryption for Zabbix proxy

1. Prepare files with top-level CA certificates, proxy certificate (chain) and private key as described in Configuring certificate on Zabbix server. Edit parameters TLSCAFile, TLSCertFile, TLSKeyFile in proxy configuration accordingly.

2. For active proxy edit TLSConnect parameter:

  1. TLSConnect=cert

For passive proxy edit TLSAccept parameter:

  1. TLSAccept=cert

3. Now you have a minimal certificate-based proxy configuration. You may prefer to improve proxy security by setting TLSServerCertIssuer and TLSServerCertSubject parameters (see Restricting allowed certificate Issuer and Subject).

4. In final proxy configuration file TLS parameters may look like:

  1. TLSConnect=cert
  2.        TLSAccept=cert
  3.        TLSCAFile=/home/zabbix/zabbix_ca_file
  4.        TLSServerCertIssuer=CN=Signing CA,OU=Development group,O=Zabbix SIA,DC=zabbix,DC=com
  5.        TLSServerCertSubject=CN=Zabbix server,OU=Development group,O=Zabbix SIA,DC=zabbix,DC=com
  6.        TLSCertFile=/home/zabbix/zabbix_proxy.crt
  7.        TLSKeyFile=/home/zabbix/zabbix_proxy.key

5. Configure encryption for this proxy in Zabbix frontend:

  • Go to: Administration → Proxies
  • Select proxy and click on Encryption tab

In examples below Issuer and Subject fields are filled in - see Restricting allowed certificate Issuer and Subject why and how to use these fields.

For active proxy

proxy_active_cert.png

For passive proxy

proxy_passive_cert.png

Configuring certificate-based encryption for Zabbix agent

1. Prepare files with top-level CA certificates, agent certificate (chain) and private key as described in Configuring certificate on Zabbix server. Edit parameters TLSCAFile, TLSCertFile, TLSKeyFile in agent configuration accordingly.

2. For active checks edit TLSConnect parameter:

  1. TLSConnect=cert

For passive checks edit TLSAccept parameter:

  1. TLSAccept=cert

3. Now you have a minimal certificate-based agent configuration. You may prefer to improve agent security by setting TLSServerCertIssuer and TLSServerCertSubject parameters. (see Restricting allowed certificate Issuer and Subject).

4. In final agent configuration file TLS parameters may look like:

  1. TLSConnect=cert
  2.        TLSAccept=cert
  3.        TLSCAFile=/home/zabbix/zabbix_ca_file
  4.        TLSServerCertIssuer=CN=Signing CA,OU=Development group,O=Zabbix SIA,DC=zabbix,DC=com
  5.        TLSServerCertSubject=CN=Zabbix proxy,OU=Development group,O=Zabbix SIA,DC=zabbix,DC=com
  6.        TLSCertFile=/home/zabbix/zabbix_agentd.crt
  7.        TLSKeyFile=/home/zabbix/zabbix_agentd.key

(Example assumes that host is monitored via proxy, hence proxy certificate Subject.)

5. Configure encryption for this agent in Zabbix frontend:

  • Go to: Data collection → Hosts
  • Select host and click on Encryption tab

In example below Issuer and Subject fields are filled in - see Restricting allowed certificate Issuer and Subject why and how to use these fields.

agent_config.png

Restricting allowed certificate Issuer and Subject

When two Zabbix components (e.g. server and agent) establish a TLS connection they both check each others certificates. If a peer certificate is signed by a trusted CA (with pre-configured top-level certificate in TLSCAFile), is valid, has not expired and passes some other checks then communication can proceed. Certificate issuer and subject are not checked in this simplest case.

Here is a risk - anybody with a valid certificate can impersonate anybody else (e.g. a host certificate can be used to impersonate server). This may be acceptable in small environments where certificates are signed by a dedicated in-house CA and risk of impersonating is low.

If your top-level CA is used for issuing other certificates which should not be accepted by Zabbix or you want to reduce risk of impersonating you can restrict allowed certificates by specifying their Issuer and Subject strings.

For example, you can write in Zabbix proxy configuration file:

  1. TLSServerCertIssuer=CN=Signing CA,OU=Development group,O=Zabbix SIA,DC=zabbix,DC=com
  2.        TLSServerCertSubject=CN=Zabbix server,OU=Development group,O=Zabbix SIA,DC=zabbix,DC=com

With these settings, an active proxy will not talk to Zabbix server with different Issuer or Subject string in certificate, a passive proxy will not accept requests from such server.

A few notes about Issuer or Subject string matching:

  1. Issuer and Subject strings are checked independently. Both are optional.
  2. UTF-8 characters are allowed.
  3. Unspecified string means any string is accepted.
  4. Strings are compared “as-is”, they must be exactly the same to match.
  5. Wildcards and regexp’s are not supported in matching.
  6. Only some requirements from RFC 4514 Lightweight Directory Access Protocol (LDAP): String Representation of Distinguished Names are implemented:
    1. escape characters ‘“‘ (U+0022), ‘+’ U+002B, ‘,’ U+002C, ‘;’ U+003B, ‘<’ U+003C, ‘>’ U+003E, ‘\‘ U+005C anywhere in string.
    2. escape characters space (‘ ‘ U+0020) or number sign (‘#’ U+0023) at the beginning of string.
    3. escape character space (‘ ‘ U+0020) at the end of string.
  7. Match fails if a null character (U+0000) is encountered (RFC 4514 allows it).
  8. Requirements of RFC 4517 Lightweight Directory Access Protocol (LDAP): Syntaxes and Matching Rules and RFC 4518 Lightweight Directory Access Protocol (LDAP): Internationalized String Preparation are not supported due to amount of work required.

Order of fields in Issuer and Subject strings and formatting are important! Zabbix follows RFC 4514 recommendation and uses “reverse” order of fields.

The reverse order can be illustrated by example:

  1. TLSServerCertIssuer=CN=Signing CA,OU=Development group,O=Zabbix SIA,DC=zabbix,DC=com
  2.        TLSServerCertSubject=CN=Zabbix proxy,OU=Development group,O=Zabbix SIA,DC=zabbix,DC=com

Note that it starts with low level (CN), proceeds to mid-level (OU, O) and ends with top-level (DC) fields.

OpenSSL by default shows certificate Issuer and Subject fields in “normal” order, depending on additional options used:

  1. $ openssl x509 -noout -in /home/zabbix/zabbix_proxy.crt -issuer -subject
  2.        issuer= /DC=com/DC=zabbix/O=Zabbix SIA/OU=Development group/CN=Signing CA
  3.        subject= /DC=com/DC=zabbix/O=Zabbix SIA/OU=Development group/CN=Zabbix proxy
  5.        $ openssl x509 -noout -text -in /home/zabbix/zabbix_proxy.crt
  6.        Certificate:
  7.                ...
  8.                Issuer: DC=com, DC=zabbix, O=Zabbix SIA, OU=Development group, CN=Signing CA
  9.            ...
  10.                Subject: DC=com, DC=zabbix, O=Zabbix SIA, OU=Development group, CN=Zabbix proxy

Here Issuer and Subject strings start with top-level (DC) and end with low-level (CN) field, spaces and field separators depend on options used. None of these values will match in Zabbix Issuer and Subject fields!

To get proper Issuer and Subject strings usable in Zabbix invoke OpenSSL with special options
-nameopt esc_2253,esc_ctrl,utf8,dump_nostr,dump_unknown,dump_der,sep_comma_plus,dn_rev,sname:

  1. $ openssl x509 -noout -issuer -subject \
  2.                -nameopt esc_2253,esc_ctrl,utf8,dump_nostr,dump_unknown,dump_der,sep_comma_plus,dn_rev,sname \
  3.                -in /home/zabbix/zabbix_proxy.crt
  4.        issuer= CN=Signing CA,OU=Development group,O=Zabbix SIA,DC=zabbix,DC=com
  5.        subject= CN=Zabbix proxy,OU=Development group,O=Zabbix SIA,DC=zabbix,DC=com

Now string fields are in reverse order, fields are comma-separated, can be used in Zabbix configuration files and frontend.

Limitations on using X.509 v3 certificate extensions

  • Subject Alternative Name (subjectAltName) extension.
    Alternative subject names from subjectAltName extension (like IP address, e-mail address) are not supported by Zabbix. Only value of “Subject” field can be checked in Zabbix (see Restricting allowed certificate Issuer and Subject).
    If certificate uses the subjectAltName extension then result depends on particular combination of crypto toolkits Zabbix components are compiled with (it may or may not work, Zabbix may refuse to accept such certificates from peers).
  • Extended Key Usage extension.
    If used then generally both clientAuth (TLS WWW client authentication) and serverAuth (TLS WWW server authentication) are necessary.
    For example, in passive checks Zabbix agent acts in a TLS server role, so serverAuth must be set in agent certificate. For active checks agent certificate needs clientAuth to be set.
    GnuTLS issues a warning in case of key usage violation but allows communication to proceed.
  • Name Constraints extension.
    Not all crypto toolkits support it. This extension may prevent Zabbix from loading CA certificates where this section is marked as critical (depends on particular crypto toolkit).

Certificate Revocation Lists (CRL)

If a certificate is compromised CA can revoke it by including in CRL. CRLs can be configured in server, proxy and agent configuration file using parameter TLSCRLFile. For example:

  1. TLSCRLFile=/home/zabbix/zabbix_crl_file

where zabbix_crl_file may contain CRLs from several CAs and look like:

  1. -----BEGIN X509 CRL-----
  2.        MIIB/DCB5QIBATANBgkqhkiG9w0BAQUFADCBgTETMBEGCgmSJomT8ixkARkWA2Nv
  3.        ...
  4.        treZeUPjb7LSmZ3K2hpbZN7SoOZcAoHQ3GWd9npuctg=
  5.        -----END X509 CRL-----
  6.        -----BEGIN X509 CRL-----
  7.        MIIB+TCB4gIBATANBgkqhkiG9w0BAQUFADB/MRMwEQYKCZImiZPyLGQBGRYDY29t
  8.        ...
  9.        CAEebS2CND3ShBedZ8YSil59O6JvaDP61lR5lNs=
  10.        -----END X509 CRL-----

CRL file is loaded only on Zabbix start. CRL update requires restart.

If Zabbix component is compiled with OpenSSL and CRLs are used then each top and intermediate level CA in certificate chains must have a corresponding CRL (it can be empty) in TLSCRLFile.