Monitoring and Alerting

The rancher-monitoring application can quickly deploy leading open-source monitoring and alerting solutions onto your cluster.

Introduced in Rancher v2.5, the application is powered by Prometheus, Grafana, Alertmanager, the Prometheus Operator, and the Prometheus adapter.

For information on V1 monitoring and alerting, available in Rancher v2.2 up to v2.4, please see the Rancher v2.0—v2.4 docs on cluster monitoring, alerting, notifiers and other tools.

Using the rancher-monitoring application, you can quickly deploy leading open-source monitoring and alerting solutions onto your cluster.

Features

Prometheus lets you view metrics from your Rancher and Kubernetes objects. Using timestamps, Prometheus lets you query and view these metrics in easy-to-read graphs and visuals, either through the Rancher UI or Grafana, which is an analytics viewing platform deployed along with Prometheus.

By viewing data that Prometheus scrapes from your cluster control plane, nodes, and deployments, you can stay on top of everything happening in your cluster. You can then use these analytics to better run your organization: stop system emergencies before they start, develop maintenance strategies, or restore crashed servers.

The monitoring application:

• Monitors the state and processes of your cluster nodes, Kubernetes components, and software deployments.
• Defines alerts based on metrics collected via Prometheus.
• Creates custom Grafana dashboards.
• Configures alert-based notifications via email, Slack, PagerDuty, etc. using Prometheus Alertmanager.
• Defines precomputed, frequently needed or computationally expensive expressions as new time series based on metrics collected via Prometheus.
• Exposes collected metrics from Prometheus to the Kubernetes Custom Metrics API via Prometheus Adapter for use in HPA.

See How Monitoring Works for an explanation of how the monitoring components work together.

Default Components and Deployments

Built-in Dashboards

By default, the monitoring application deploys Grafana dashboards (curated by the kube-prometheus project) onto a cluster.

It also deploys an Alertmanager UI and a Prometheus UI. For more information about these tools, see Built-in Dashboards.

Default Metrics Exporters

By default, Rancher Monitoring deploys exporters (such as node-exporter and kube-state-metrics).

These default exporters automatically scrape metrics for CPU and memory from all components of your Kubernetes cluster, including your workloads.

Default Alerts

The monitoring application deploys some alerts by default. To see the default alerts, go to the Alertmanager UI and click Expand all groups.

Components Exposed in the Rancher UI

For a list of monitoring components exposed in the Rancher UI, along with common use cases for editing them, see this section.

Role-based Access Control

For more information on configuring access to monitoring, see this page.

note

Rancher and Project read permissions don’t necessarily apply to monitoring resources. See monitoring-ui-view for more details.

Guides

• Enable monitoring
• Uninstall monitoring
• Monitoring workloads
• Customizing Grafana dashboards
• Persistent Grafana dashboards
• Debugging high memory usage

Configuration

Configuring Monitoring Resources in Rancher

The configuration reference assumes familiarity with how monitoring components work together. For more information, see How Monitoring Works.

• ServiceMonitor and PodMonitor
• Receiver
• Route
• PrometheusRule
• Prometheus
• Alertmanager

Configuring Helm Chart Options

For more information on rancher-monitoring chart options, including options to set resource limits and requests, see Helm Chart Options.

Windows Cluster Support

When deployed onto an RKE1 Windows cluster, Monitoring V2 will now automatically deploy a windows-exporter DaemonSet and set up a ServiceMonitor to collect metrics from each of the deployed Pods. This will populate Prometheus with windows_ metrics that are akin to the node_ metrics exported by node_exporter for Linux hosts.

To be able to fully deploy Monitoring V2 for Windows, all of your Windows hosts must have a minimum wins version of v0.1.0.

For more details on how to upgrade wins on existing Windows hosts, see Windows cluster support for Monitoring V2..

Known Issues

There is a known issue that K3s clusters require more than the allotted default memory. If you enable monitoring on a K3s cluster, set prometheus.prometheusSpec.resources.memory.limit to 2500 Mi and prometheus.prometheusSpec.resources.memory.request to 1750 Mi.

See Debugging High Memory Usage for advice and recommendations.