Basic Configuration
Advanced Options
- Prometheus RemoteRead and RemoteWrite
- LivenessProbe and ReadinessProbe
Node Exporter
Persistent Storage
Remote Storage

Available as of v2.2.0

While configuring monitoring at either the cluster level or project level, there are multiple options that can be configured.

Basic Configuration
Advanced Options
Node Exporter
Persistent Storage
Remote Storage

Basic Configuration

Option	Description
Data Retention	How long your Prometheus instance retains monitoring data scraped from Rancher objects before it’s purged.
Enable Node Exporter	Whether or not to deploy the node exporter.
Node Exporter Host Port	The host port on which data is exposed, i.e. data that Prometheus collects from your node hardware. Required if you have enabled the node exporter.
Enable Persistent Storage for Prometheus	Whether or not to configure storage for Prometheus so that metrics can be retained even if the Prometheus pod fails.
Enable Persistent Storage for Grafana	Whether or not to configure storage for Grafana so that the Grafana dashboards and configuration can be retained even if the Grafana pod fails.
Prometheus CPU Limit	CPU resource limit for the Prometheus pod.
Prometheus CPU Reservation	CPU reservation for the Prometheus pod.
Prometheus Memory Limit	Memory resource limit for the Prometheus pod.
Prometheus Memory Reservation	Memory resource requests for the Prometheus pod.
Selector	Ability to select the nodes in which Prometheus and Grafana pods are deployed to. To use this option, the nodes must have labels.

Advanced Options

Since monitoring is an application from the Rancher catalog, it can be configured like any other catalog application, by passing in values to Helm.

Warning: Any modification to the application without understanding the entire application can lead to catastrophic errors.

Prometheus RemoteRead and RemoteWrite

Available as of v2.4.0

Prometheus RemoteRead and RemoteWrite can be configured as custom answers in the Advanced Options section.

For more information on remote endpoints and storage, refer to the Prometheus documentation.

The Prometheus operator documentation contains the full RemoteReadSpec and RemoteWriteSpec.

An example configuration would be:

Variable	Value
`prometheus.remoteWrite[0].url`	`http://mytarget.com`

LivenessProbe and ReadinessProbe

Available as of v2.4.0

Prometheus LivenessProbe and ReadinessProbe can be configured as custom answers in the Advanced Options section.

The Kubernetes probe spec is here.

Some example key-value pairs are:

Variable	Value
`prometheus.livenessProbe.timeoutSeconds`	60
`prometheus.readinessProbe.timeoutSeconds`	60

Node Exporter

The node exporter is a popular open source exporter, which exposes the metrics for hardware and *NIX kernels OS. It is designed to monitor the host system. However, there are still issues with namespaces when running it in a container, mostly around filesystem mount spaces. In order to monitor actual network metrics for the container network, the node exporter must be deployed with the hostNetwork mode.

When configuring Prometheus and enabling the node exporter, enter a host port in the Node Exporter Host Port that will not produce port conflicts with existing applications. The host port chosen must be open to allow internal traffic between Prometheus and the Node Exporter.

Warning: In order for Prometheus to collect the metrics of the node exporter, after enabling cluster monitoring, you must open the Node Exporter Host Port in the host firewall rules to allow intranet access. By default, 9796 is used as that host port.

Persistent Storage

Prerequisite: Configure one or more StorageClasses to use as persistent storage for your Prometheus or Grafana pod.

By default, when you enable Prometheus for either a cluster or project, all monitoring data that Prometheus collects is stored on its own pod. With local storage, if the Prometheus or Grafana pods fail, all the data is lost. Rancher recommends configuring an external persistent storage to the cluster. With the external persistent storage, if the Prometheus or Grafana pods fail, the new pods can recover using data from the persistent storage.

When enabling persistent storage for Prometheus or Grafana, specify the size of the persistent volume and select the StorageClass.

Remote Storage

Prerequisite: Need a remote storage endpoint to be available. The possible list of integrations is available here

Using advanced options, remote storage integration for the Prometheus installation can be configured as follows:

prometheus.remoteWrite[0].url = http://remote1/push
prometheus.remoteWrite[0].remoteTimeout = 33s
prometheus.remoteWrite[1].url = http://remote2/push
prometheus.remoteRead[0].url = http://remote1/read
prometheus.remoteRead[0].proxyUrl = http://proxy.url
prometheus.remoteRead[0].bearerToken = token-value
prometheus.remoteRead[1].url = http://remote2/read
prometheus.remoteRead[1].remoteTimeout = 33s
prometheus.remoteRead[1].readRecent = true

Additional fields can be set up based on the ReadSpec and RemoteWriteSpec