Manage Multi-Tenant Notifications with Notification Manager

Notification Manager manages notifications in KubeSphere. It receives alerts or notifications from different senders and then sends notifications to different users.

Supported senders include:

  • Prometheus Alertmanager
  • Custom sender (Coming soon)

Supported receivers include:

notification-manager

Quickstart

Configure Prometheus Alertmanager to send alerts to Notification Manager

Notification Manager uses the port 19093 and API path /api/v2/alerts to receive alerts sent from Prometheus Alertmanager of KubeSphere.

To receive Alertmanager alerts, KubeSphere already added the Alertmanager webhook and route configurations like below (by editing the Secret alertmanager-main in the namespace kubesphere-monitoring-system):

Send Prometheus alerts to Notification Manager:

  1.     "receivers":
  2.     - "name": "prometheus"
  3.       "webhook_configs":
  4.       - "url": "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"
  5.     "route":
  6.       "routes":
  7.       - "match":
  8.           "alerttype": ""
  9.         "receiver": "prometheus"

Send event alerts to Notification Manager:

  1.     "receivers":
  2.     - "name": "event"
  3.       "webhook_configs":
  4.       - "url": "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"
  5.         "send_resolved": false
  6.     "route":
  7.       "routes":
  8.       - "match":
  9.           "alerttype": "event"
  10.         "receiver": "event"
  11.         "group_interval": "30s"

Send auditing alerts to Notification Manager:

  1.     "receivers":
  2.     - "name": "auditing"
  3.       "webhook_configs":
  4.       - "url": "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"
  5.         "send_resolved": false
  6.     "route":
  7.       "routes":
  8.       - "match":
  9.           "alerttype": "auditing"
  10.         "receiver": "auditing"
  11.         "group_interval": "30s"

Note

The above is the default configuration. If you do not want to receive a certain type of alert, you can delete the corresponding configuration.

Configure receivers

Notification Manager now supports three types of receivers: Email, WeChat Work and Slack. Only the administrator can configure receivers.

Email

If a tenant named test-user who wants to receive email notifications, create an email receiver as follows:

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: v1
  3. data:
  4.   password: dGVzdA==
  5. kind: Secret
  6. metadata:
  7.   labels:
  8.     app: notification-manager
  9.   name: test-user-email-secret
  10.   namespace: kubesphere-monitoring-system
  11. type: Opaque
  13. ---
  14. apiVersion: notification.kubesphere.io/v1alpha1
  15. kind: EmailConfig
  16. metadata:
  17.   labels:
  18.     app: notification-manager
  19.     type: tenant
  20.     user: test-user
  21.   name: test-user-config
  22.   namespace: kubesphere-monitoring-system
  23. spec:
  24.   authPassword:
  25.     key: password
  26.     name: test-user-email-secret
  27.   authUsername: abc1
  28.   from: [email protected]
  29.   requireTLS: true
  30.   smartHost:
  31.     host: imap.xyz.com
  32.     port: "25"
  34. ---
  35. apiVersion: notification.kubesphere.io/v1alpha1
  36. kind: EmailReceiver
  37. metadata:
  38.   labels:
  39.     app: notification-manager
  40.     type: tenant
  41.     user: test-user
  42.   name: test-user-receiver
  43.   namespace: kubesphere-monitoring-system
  44. spec:
  45.   emailConfigSelector:
  46.     matchLabels:
  47.       type: tenant
  48.       user: test-user
  49.   to:
  50.   - [email protected]
  51.   - [email protected]
  52. EOF

emailConfigSelector is a selector to select EmailConfig for the email receiver. If emailConfigSelector is not set, the receiver will use the default email configuration. You can create a default email configuration as follows:

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: v1
  3. data:
  4.   password: dGVzdA==
  5. kind: Secret
  6. metadata:
  7.   labels:
  8.     app: notification-manager
  9.   name: default-email-secret
  10.   namespace: kubesphere-monitoring-system
  11. type: Opaque
  13. ---
  14. apiVersion: notification.kubesphere.io/v1alpha1
  15. kind: EmailConfig
  16. metadata:
  17.   labels:
  18.     app: notification-manager
  19.     type: default
  20.   name: default-email-config
  21.   namespace: kubesphere-monitoring-system
  22. spec:
  23.   authPassword:
  24.     key: password
  25.     name: default-email-secret
  26.   authUsername: default
  27.   from: [email protected]
  28.   requireTLS: true
  29.   smartHost:
  30.     host: imap.xyz.com
  31.     port: "25"
  32. EOF

Email receivers with the label type: tenant only receive notifications from the namespace to which the specified tenant user has access. If you want them to receive notifications from all namespaces or even without a namespace label, you can create a global email receiver with the label type: global as below:

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: notification.kubesphere.io/v1alpha1
  3. kind: EmailReceiver
  4. metadata:
  5.   labels:
  6.     app: notification-manager
  7.     type: global
  8.   name: global-email-receiver
  9.   namespace: kubesphere-monitoring-system
  10. spec:
  11.   to:
  12.   - [email protected]
  13. EOF

Note

The global email receiver will use the default email configuration.

WeChat Work

Notification Manager supports sending notifications to WeChat Work. If a tenant named test-user who wants to receive notifications from WeChat Work, create a WeChat receiver as follows:

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: v1
  3. data:
  4.   wechat: dGVzdA==
  5. kind: Secret
  6. metadata:
  7.   labels:
  8.     app: notification-manager
  9.   name: test-user-wechat-secret
  10.   namespace: kubesphere-monitoring-system
  11. type: Opaque
  13. ---
  14. apiVersion: notification.kubesphere.io/v1alpha1
  15. kind: WechatConfig
  16. metadata:
  17.   name: test-user-config
  18.   namespace: kubesphere-monitoring-system
  19.   labels:
  20.     app: notification-manager
  21.     type: tenant
  22.     user: test-user
  23. spec:
  24.   wechatApiUrl: https://qyapi.weixin.qq.com/cgi-bin/
  25.   wechatApiSecret:
  26.     key: wechat
  27.     name: test-user-wehat-secret
  28.   wechatApiCorpId: wwfd76b24f06513578
  29.   wechatApiAgentId: "1000002"
  31. ---
  32. apiVersion: notification.kubesphere.io/v1alpha1
  33. kind: WechatReceiver
  34. metadata:
  35.   name: test-user-wechat
  36.   namespace: kubesphere-monitoring-system
  37.   labels:
  38.     app: notification-manager
  39.     type: tenant
  40.     user: test-user
  41. spec:
  42.   wechatConfigSelector:
  43.     matchLabels:
  44.       type: tenant
  45.       user: test-user
  46.   # optional
  47.   # One of toUser, toParty, toParty should be specified.
  48.   toUser: user1 | user2
  49.   toParty: party1 | party2
  50.   toTag: tag1 | tag2
  51. EOF

Info

  • wechatApiCorpId is the id of your WeChat Work.
  • wechatApiAgentId is the id of the app sending messages to users in your WeChat Work.
  • wechatApiSecret is the secret of this app. You can get these two parameters in App Management of your WeChat Work.
  • Any user, party or tag who wants to receive notifications must be in the allowed users list of this app.

wechatConfigSelector is a selector to select WechatConfig for the WeChat receiver. If wechatConfigSelector is not set, the WeChat receiver will use the default WeChat configuration. You can create a default WeChat configuration as follows:

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: v1
  3. data:
  4.   wechat: dGVzdA==
  5. kind: Secret
  6. metadata:
  7.   labels:
  8.     app: notification-manager
  9.   name: default-wechat-secret
  10.   namespace: kubesphere-monitoring-system
  11. type: Opaque
  13. ---
  14. apiVersion: notification.kubesphere.io/v1alpha1
  15. kind: WechatConfig
  16. metadata:
  17.   name: default-wechat-config
  18.   namespace: kubesphere-monitoring-system
  19.   labels:
  20.     app: notification-manager
  21.     type: default
  22. spec:
  23.   wechatApiUrl: https://qyapi.weixin.qq.com/cgi-bin/
  24.   wechatApiSecret:
  25.     key: wechat
  26.     name: default-wechat-secret
  27.   wechatApiCorpId: wwfd76b24f06513578
  28.   wechatApiAgentId: "1000002"
  29. EOF

WeChat receivers with the label type: tenant can only receive notifications from the namespace to which the specified tenant user has access. If you want them to receive notifications from all namespaces or even without a namespace label, you can create a global WeChat receiver with the label type: global as below:

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: notification.kubesphere.io/v1alpha1
  3. kind: WechatReceiver
  4. metadata:
  5.   name: global-wechat-wechat
  6.   namespace: kubesphere-monitoring-system
  7.   labels:
  8.     app: notification-manager
  9.     type: global
  10. spec:
  11.   # optional
  12.   # One of toUser, toParty, toParty should be specified.
  13.   toUser: global
  14.   toParty: global
  15.   toTag: global
  16. EOF

Note

The global WeChat receiver will use the default WeChat configuration.

Slack

Notification Manager supports sending notifications to Slack channels. If a tenant named test-user who wants to receive notifications from Slack, create a Slack receiver as follows:

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: v1
  3. data:
  4.   token: dGVzdA==
  5. kind: Secret
  6. metadata:
  7.   labels:
  8.     app: notification-manager
  9.   name: test-user-slack-secret
  10.   namespace: kubesphere-monitoring-system
  11. type: Opaque
  13. ---
  14. apiVersion: notification.kubesphere.io/v1alpha1
  15. kind: SlackConfig
  16. metadata:
  17.   name: test-user-config
  18.   namespace: kubesphere-monitoring-system
  19.   labels:
  20.     app: notification-manager
  21.     type: tenant
  22.     user: test-user
  23. spec:
  24.   slackTokenSecret: 
  25.     key: token
  26.     name: test-user-slack-secret
  28. ---
  29. apiVersion: notification.kubesphere.io/v1alpha1
  30. kind: SlackReceiver
  31. metadata:
  32.   name: test-user-slack
  33.   namespace: kubesphere-monitoring-system
  34.   labels:
  35.     app: notification-manager
  36.     type: tenant
  37.     user: test-user
  38. spec:
  39.   slackConfigSelector:
  40.     matchLabels:
  41.       type: tenant
  42.       user: test-user
  43.   channel: alert
  44. EOF

Info

  • The Slack token is the OAuth Access Token or Bot User OAuth Access Token when you create a Slack app.
  • This app must have the scope chat:write.
  • The user who creates the app or bot user must be in the channel to which you want to send notifications.

slackConfigSelector is a selector to select SlackConfig for the Slack receiver. If slackConfigSelector is not set, the Slack receiver will use the default Slack configuration. You can create a default Slack configuration as follows:

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: v1
  3. data:
  4.   token: dGVzdA==
  5. kind: Secret
  6. metadata:
  7.   labels:
  8.     app: notification-manager
  9.   name: default-slack-secret
  10.   namespace: kubesphere-monitoring-system
  11. type: Opaque
  13. ---
  14. apiVersion: notification.kubesphere.io/v1alpha1
  15. kind: SlackConfig
  16. metadata:
  17.   name: default-slack-config
  18.   namespace: kubesphere-monitoring-system
  19.   labels:
  20.     app: notification-manager
  21.     type: default
  22. spec:
  23.   slackTokenSecret: 
  24.     key: token
  25.     name: default-slack-secret
  26. EOF

Slack receivers with the label type: tenant can only receive notifications from the namespace to which the specified tenant user has access. If you want them to receive notifications from all namespaces or even without a namespace label, you can create a global Slack receiver with the label type: global as below:

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: notification.kubesphere.io/v1alpha1
  3. kind: SlackReceiver
  4. metadata:
  5.   name: global-slack-slack
  6.   namespace: kubesphere-monitoring-system
  7.   labels:
  8.     app: notification-manager
  9.     type: global
  10. spec:
  11.   channel: global
  12. EOF

Note

The global Slack receiver will use the default Slack configuration.