Configuring alert notifications

In OKD, an alert is fired when the conditions defined in an alerting rule are true. An alert provides a notification that a set of circumstances are apparent within a cluster. Firing alerts can be viewed in the Alerting UI in the OKD web console by default. After an installation, you can configure OKD to send alert notifications to external systems.

Sending notifications to external systems

In OKD 4.9, firing alerts can be viewed in the Alerting UI. Alerts are not configured by default to be sent to any notification systems. You can configure OKD to send alerts to the following receiver types:

  • PagerDuty

  • Webhook

  • Email

  • Slack

Routing alerts to receivers enables you to send timely notifications to the appropriate teams when failures occur. For example, critical alerts require immediate attention and are typically paged to an individual or a critical response team. Alerts that provide non-critical warning notifications might instead be routed to a ticketing system for non-immediate review.

Checking that alerting is operational by using the watchdog alert

OKD monitoring includes a watchdog alert that fires continuously. Alertmanager repeatedly sends watchdog alert notifications to configured notification providers. The provider is usually configured to notify an administrator when it stops receiving the watchdog alert. This mechanism helps you quickly identify any communication issues between Alertmanager and the notification provider.

Configuring alert receivers

You can configure alert receivers to ensure that you learn about important issues with your cluster.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. In the Administrator perspective, navigate to Administration → Cluster Settings → Configuration → Alertmanager.

    Alternatively, you can navigate to the same page through the notification drawer. Select the bell icon at the top right of the OKD web console and choose Configure in the AlertmanagerReceiverNotConfigured alert.

  2. Select Create Receiver in the Receivers section of the page.

  3. In the Create Receiver form, add a Receiver Name and choose a Receiver Type from the list.

  4. Edit the receiver configuration:

    • For PagerDuty receivers:

      1. Choose an integration type and add a PagerDuty integration key.

      2. Add the URL of your PagerDuty installation.

      3. Select Show advanced configuration if you want to edit the client and incident details or the severity specification.

    • For webhook receivers:

      1. Add the endpoint to send HTTP POST requests to.

      2. Select Show advanced configuration if you want to edit the default option to send resolved alerts to the receiver.

    • For email receivers:

      1. Add the email address to send notifications to.

      2. Add SMTP configuration details, including the address to send notifications from, the smarthost and port number used for sending emails, the hostname of the SMTP server, and authentication details.

      3. Choose whether TLS is required.

      4. Select Show advanced configuration if you want to edit the default option not to send resolved alerts to the receiver or edit the body of email notifications configuration.

    • For Slack receivers:

      1. Add the URL of the Slack webhook.

      2. Add the Slack channel or user name to send notifications to.

      3. Select Show advanced configuration if you want to edit the default option not to send resolved alerts to the receiver or edit the icon and username configuration. You can also choose whether to find and link channel names and usernames.

  1. By default, firing alerts with labels that match all of the selectors will be sent to the receiver. If you want label values for firing alerts to be matched exactly before they are sent to the receiver:

    1. Add routing label names and values in the Routing Labels section of the form.

    2. Select Regular Expression if want to use a regular expression.

    3. Select Add Label to add further routing labels.

  2. Select Create to create the receiver.

Additional resources