Overview of Grafana 8 alerts

Alerts allow you to know about problems in your systems moments after they occur. Robust and actionable alerts help you identify and resolve issues quickly, minimizing disruption to your services.

Note: This information is for the new, Grafana 8 Alerts. This is an opt-in feature released in Grafana 8.0. Grafana still supports legacy dashboard alerts out of the box

Alerts have four main components:

• Alerting rule - One or more query and/or expression, a condition, the frequency of evaluation, and the (optional) duration that a condition must be met before creating an alert.
• Contact point - A channel for sending notifications when the conditions of an alerting rule are met.
• Notification policy - A set of matching and grouping criteria used to determine where, and how frequently, to send notifications.
• Silences - Date and matching criteria used to silence notifications.

Alerting tasks

You can perform the following tasks for alerts:

• Create a Grafana managed alert rule
• Create a Cortex or Loki managed alert rule
• View existing alert rules and their current state
• Test alert rules and troubleshoot
• Add or edit an alert contact point
• Add or edit notification policies
• Create and edit silences

Clustering

Currently alerting supports a limited form of high availability. Alert notifications are deduped when running multiple servers. This means all alerts are executed on every server but no duplicate alert notifications are sent due to the deduping logic. Proper load balancing of alerts will be introduced in the future.

Alert evaluation

Grafana managed alerts are evaluated by the Grafana backend. Rule evaluations are scheduled, according to the alert rule configuration, and queries are evaluated by an engine that is part of core Grafana.

Alerting rules can only query backend data sources with alerting enabled:

• builtin or developed and maintained by grafana: Graphite, Prometheus, Loki, InfluxDB, Elasticsearch, Google Cloud Monitoring, Cloudwatch, Azure Monitor, MySQL, PostgreSQL, MSSQL, OpenTSDB, Oracle, and Azure Data Explorer
• any community backend data sources with alerting enabled (backend and alerting properties are set in the plugin.json)

Metrics from the alerting engine

The alerting engine publishes some internal metrics about itself. You can read more about how Grafana publishes internal metrics.

• View alert rules and their current state