Troubleshoot alerts

If alerts are not behaving as you expect, here are some steps you can take to troubleshoot and figure out what is going wrong.

The first level of troubleshooting you can do is click Test Rule. You will get result back that you can expand to the point where you can see the raw data that was returned from your query.

Further troubleshooting can also be done by inspecting the grafana-server log. If it’s not an error or for some reason the log does not say anything you can enable debug logging for some relevant components. This is done in Grafana’s ini config file.

Example showing loggers that could be relevant when troubleshooting alerting.

  1. [log]
  2. filters = alerting.scheduler:debug \
  3.           alerting.engine:debug \
  4.           alerting.resultHandler:debug \
  5.           alerting.evalHandler:debug \
  6.           alerting.evalContext:debug \
  7.           alerting.extractor:debug \
  8.           alerting.notifier:debug \
  9.           alerting.notifier.slack:debug \
  10.           alerting.notifier.pagerduty:debug \
  11.           alerting.notifier.email:debug \
  12.           alerting.notifier.webhook:debug \
  13.           tsdb.graphite:debug \
  14.           tsdb.prometheus:debug \
  15.           tsdb.opentsdb:debug \
  16.           tsdb.influxdb:debug \
  17.           tsdb.elasticsearch:debug \
  18.           tsdb.elasticsearch.client:debug \

If you want to log raw query sent to your TSDB and raw response in log you also have to set grafana.ini option app_mode to development.