Rancher can integrate with a variety of popular logging services and tools that exist outside of your Kubernetes clusters.

For background information about how logging integrations work, refer to the cluster administration section.

Rancher supports the following services:

• Elasticsearch
• Splunk
• Kafka
• Syslog
• Fluentd

Note: You can only configure one logging service per cluster or per project.

Only administrators, cluster owners or members, or project owners can configure Rancher to send Kubernetes logs to a logging service.

Requirements

The Docker daemon on each node in the cluster should be configured with the (default) log-driver: json-file. You can check the log-driver by running the following command:

$ docker info | grep 'Logging Driver'
Logging Driver: json-file

Advantages

Setting up a logging service to collect logs from your cluster/project has several advantages:

• Logs errors and warnings in your Kubernetes infrastructure to a stream. The stream informs you of events like a container crashing, a pod eviction, or a node dying.
• Allows you to capture and analyze the state of your cluster and look for trends in your environment using the log stream.
• Helps you when troubleshooting or debugging.
• Saves your logs to a safe location outside of your cluster, so that you can still access them even if your cluster encounters issues.

Logging Scope

You can configure logging at either cluster level or project level.

• Cluster logging writes logs for every pod in the cluster, i.e. in all the projects. For RKE clusters, it also writes logs for all the Kubernetes system components.

• Project logging writes logs for every pod in that particular project.

Logs that are sent to your logging service are from the following locations:

• Pod logs stored at /var/log/containers.

• Kubernetes system components logs stored at /var/lib/rancher/rke/logs/.

Enabling Project Logging

1. From the Global view, navigate to the project that you want to configure project logging.

2. Select Tools > Logging in the navigation bar. In versions before v2.2.0, you can choose Resources > Logging.

3. Select a logging service and enter the configuration. Refer to the specific service for detailed configuration. Rancher supports the following services:

   • Elasticsearch
   • Splunk
   • Kafka
   • Syslog
   • Fluentd

4. (Optional) Instead of using the UI to configure the logging services, you can enter custom advanced configurations by clicking on Edit as File, which is located above the logging targets. This link is only visible after you select a logging service.

   • With the file editor, enter raw fluentd configuration for any logging service. Refer to the documentation for each logging service on how to setup the output configuration.

     • Elasticsearch Documentation
     • Splunk Documentation
     • Kafka Documentation
     • Syslog Documentation
     • Fluentd Documentation

   • If the logging service is using TLS, you also need to complete the SSL Configuration form.

     1. Provide the Client Private Key and Client Certificate. You can either copy and paste them or upload them by using the Read from a file button.

        • You can use either a self-signed certificate or one provided by a certificate authority.

        • You can generate a self-signed certificate using an openssl command. For example:

          openssl req -x509 -newkey rsa:2048 -keyout myservice.key -out myservice.cert -days 365 -nodes -subj "/CN=myservice.example.com"

     2. If you are using a self-signed certificate, provide the CA Certificate PEM.

5. (Optional) Complete the Additional Logging Configuration form.

   1. Optional: Use the Add Field button to add custom log fields to your logging configuration. These fields are key value pairs (such as foo=bar) that you can use to filter the logs from another system.

   2. Enter a Flush Interval. This value determines how often Fluentd flushes data to the logging server. Intervals are measured in seconds.

   3. Include System Log. The logs from pods in system project and RKE components will be sent to the target. Uncheck it to exclude the system logs.

6. Click Test. Rancher sends a test log to the service.

   Note: This button is replaced with Dry Run if you are using the custom configuration editor. In this case, Rancher calls the fluentd dry run command to validate the configuration.

7. Click Save.

Result: Rancher is now configured to send logs to the selected service. Log into the logging service so that you can start viewing the logs.

Related Links

Logging Architecture