Rancher Integration with Logging Services

The Logging operator now powers Rancher’s logging solution in place of the former, in-house solution.

Enabling Logging

You can enable the logging for a Rancher managed cluster by going to the Apps page and installing the logging app.

  1. Go to the cluster where you want to install logging and click Apps.
  2. Click the Logging app.
  3. Scroll to the bottom of the Helm chart README and click Install.

Result: The logging app is deployed in the cattle-logging-system namespace.

Uninstall Logging

  1. Go to the cluster where you want to install logging and click Apps.
  2. Click Installed Apps.
  3. Go to the cattle-logging-system namespace and check the boxes for rancher-logging and rancher-logging-crd.
  4. Click Delete.
  5. Confirm Delete.

Result rancher-logging is uninstalled.

Architecture

For more information about how the logging application works, see this section.

Role-based Access Control

Rancher logging has two roles, logging-admin and logging-view. For more information on how and when to use these roles, see this page.

Configuring Logging Custom Resources

To manage Flows, ClusterFlows, Outputs, and ClusterOutputs,

  1. In the upper left corner, click ☰ > Cluster Management.
  2. On the Clusters page, go to the cluster where you want to configure logging custom resources and click Explore.
  3. In the left navigation bar, click Logging.

Flows and ClusterFlows

For help with configuring Flows and ClusterFlows, see this page.

Outputs and ClusterOutputs

For help with configuring Outputs and ClusterOutputs, see this page.

Configuring the Logging Helm Chart

For a list of options that can be configured when the logging application is installed or upgraded, see this page.

Windows Support

You can enable logging from Windows nodes.

Working with a Custom Docker Root Directory

For details on using a custom Docker root directory, see this section.

Working with Taints and Tolerations

For information on how to use taints and tolerations with the logging application, see this page.

Logging V2 with SELinux

For information on enabling the logging application for SELinux-enabled nodes, see this section.

Additional Logging Sources

By default, Rancher collects logs for control plane components and node components for all cluster types. In some cases additional logs can be collected. For details, see this section.

Troubleshooting

The Logging Buffer Overloads Pods

Depending on your configuration, the default buffer size may be too large and cause pod failures. One way to reduce the load is to lower the logger’s flush interval. This prevents logs from overfilling the buffer. You can also add more flush threads to handle moments when many logs are attempting to fill the buffer at once.

For a more complete description of how to configure the logging buffer to suit your organization’s needs, see the official Logging operator documentation on buffers and on Fluentd configuration.

The cattle-logging Namespace Being Recreated

If your cluster previously deployed logging from the global view in the legacy Rancher UI, you may encounter an issue where its cattle-logging namespace is continually being recreated.

The solution is to delete all clusterloggings.management.cattle.io and projectloggings.management.cattle.io custom resources from the cluster specific namespace in the management cluster. The existence of these custom resources causes Rancher to create the cattle-logging namespace in the downstream cluster if it does not exist.

The cluster namespace matches the cluster ID, so we need to find the cluster ID for each cluster.

  1. In the upper left corner, click ☰ > Cluster Management.
  2. On the Clusters page, go to the cluster you want to get the ID of and click Explore.
  3. Copy the <cluster-id> portion from one of the URLs below. The <cluster-id> portion is the cluster namespace name.
  1. # Cluster Management UI
  2. https://<your-url>/c/<cluster-id>/
  4. # Cluster Dashboard
  5. https://<your-url>/dashboard/c/<cluster-id>/

Now that we have the <cluster-id> namespace, we can delete the CRs that cause cattle-logging to be continually recreated. Warning: ensure that logging, the version installed from the global view in the legacy Rancher UI, is not currently in use.

  1. kubectl delete crd clusterloggings.management.cattle.io -n <cluster-id>
  2. kubectl delete crd projectloggings.management.cattle.io -n <cluster-id>