使用通知脚本自定义发送告警消息
上一节介绍的回调推送方式,是一种极其灵活的推送方式,比如 FlashDuty 就是使用这种方式和夜莺对接,但是 Webhook 的方式没法复用夜莺内置的告警模板,这一节介绍另一种极其灵活的推送方式:通知脚本。
夜莺产生事件之后,会调用你指定的通知脚本,把告警事件的详细内容通过 stdin 的方式传给脚本,你就可以在脚本里做各种自定义的逻辑了。何为 stdin,请自行 Google。和 Webhook 方式类似,夜莺也会把告警事件序列化为 JSON 格式,不同的是,JSON 中不但会包含事件原始信息,还会包含事件+通知模板渲染出来的结果文本,这样就无需在脚本里处理这个模板渲染逻辑了。
菜单入口:告警通知
- 通知设置
- 通知脚本
,如下所示:
如果你要启用这个功能,需要做如下配置:
- 启用:打开
- 超时:配置一个脚本超时时间,比如 10,表示 10s 超时,千万不能设置为 0
然后就是告诉夜莺脚本内容是什么,有两个方式,一个是直接在页面上填写脚本内容,另一个是告诉夜莺脚本路径是什么,夜莺会去这个路径读取脚本内容。如果是脚本路径的方式,需要确保脚本提前放到夜莺进程所在的机器上,而且夜莺进程有读取权限,而且脚本自身要有可执行权限。
这里 放了几个 notify 脚本,可以参考。比如 notify_feishu.py 这个脚本,这是刚开始夜莺还没有内置支持飞书的时候写的,现在夜莺已经在 Go 代码里支持飞书了,所以这个脚本已经不再需要了,但是可以作为参考。其内容如下:
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
import sys
import json
import requests
class Sender(object):
@classmethod
def send_email(cls, payload):
# already done in go code
pass
@classmethod
def send_wecom(cls, payload):
# already done in go code
pass
@classmethod
def send_dingtalk(cls, payload):
# already done in go code
pass
@classmethod
def send_ifeishu(cls, payload):
users = payload.get('event').get("notify_users_obj")
tokens = {}
phones = {}
for u in users:
if u.get("phone"):
phones[u.get("phone")] = 1
contacts = u.get("contacts")
if contacts.get("feishu_robot_token", ""):
tokens[contacts.get("feishu_robot_token", "")] = 1
headers = {
"Content-Type": "application/json;charset=utf-8",
"Host": "open.feishu.cn"
}
for t in tokens:
url = "https://open.feishu.cn/open-apis/bot/v2/hook/{}".format(t)
body = {
"msg_type": "text",
"content": {
"text": payload.get('tpls').get("feishu", "feishu not found")
},
"at": {
"atMobiles": list(phones.keys()),
"isAtAll": False
}
}
response = requests.post(url, headers=headers, data=json.dumps(body))
print(f"notify_ifeishu: token={t} status_code={response.status_code} response_text={response.text}")
@classmethod
def send_mm(cls, payload):
# already done in go code
pass
@classmethod
def send_sms(cls, payload):
pass
@classmethod
def send_voice(cls, payload):
pass
def main():
payload = json.load(sys.stdin)
with open(".payload", 'w') as f:
f.write(json.dumps(payload, indent=4))
for ch in payload.get('event').get('notify_channels'):
send_func_name = "send_{}".format(ch.strip())
if not hasattr(Sender, send_func_name):
print("function: {} not found", send_func_name)
continue
send_func = getattr(Sender, send_func_name)
send_func(payload)
def hello():
print("hello nightingale")
if __name__ == "__main__":
if len(sys.argv) == 1:
main()
elif sys.argv[1] == "hello":
hello()
else:
print("I am confused")
这个脚本用到了 Python requests 库,需要自行安装。脚本核心逻辑在 main 方法中,我们重点看一下这几行代码:
def main():
payload = json.load(sys.stdin)
with open(".payload", 'w') as f:
f.write(json.dumps(payload, indent=4))
for ch in payload.get('event').get('notify_channels'):
send_func_name = "send_{}".format(ch.strip())
if not hasattr(Sender, send_func_name):
print("function: {} not found", send_func_name)
continue
send_func = getattr(Sender, send_func_name)
send_func(payload)
json.load(sys.stdin)
是从 stdin 中读取 JSON 数据,这个 JSON 数据就是告警事件的详细内容。然后把 stdin 里读取的内容写到了 .payload 文件中了,后面可以从 .payload 文件中看到具体 json 长什么样。
然后遍历 notify_channels
,这个字段是告警规则里配置的通知媒介,比如 email
、wecom
、dingtalk
、feishu
、mm
、sms
、voice
等。然后拼接成 send_
开头的函数名,比如 send_email
、send_wecom
、send_dingtalk
、send_feishu
、send_mm
、send_sms
、send_voice
等。然后用 Python 的反射机制,调用这个函数,把告警事件的详细内容传给这个函数。
然后具体的发送逻辑就在各个 send 函数中实现了,比如 send_feishu
函数中,就是调用飞书的 Webhook 接口,把告警事件的详细内容传给飞书。
实际上,你也可以自定义通知媒介,菜单入口:告警通知
- 通知设置
- 通知媒介
,然后配合自定义通知模板、user 的自定义联系方式、以及自定义通知脚本,就可以完成自定义通知逻辑了。可以据此实现电话、短信通知,甚至是自定义的 IM 通知。当然,这整个流程有点复杂,需要你对整个原理非常清楚。还不如直接使用 FlashDuty 做通知,省心一万倍。
最后,我给大家贴一下 .payload 中的内容,这个数据格式也是社区咨询比较多的,样例如下:
{
"event": {
"id": 20,
"cate": "prometheus",
"cluster": "xxx",
"datasource_id": 1,
"group_id": 1,
"group_name": "Default Busi Group",
"hash": "1951d1a34a6f1be4697269268a1c257f",
"rule_id": 6,
"rule_name": "\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22",
"rule_note": "",
"rule_prod": "metric",
"rule_algo": "",
"severity": 2,
"prom_for_duration": 0,
"prom_ql": "mem_available_percent < 100",
"rule_config": {
"queries": [
{
"keys": {
"labelKey": "",
"valueKey": ""
},
"prom_ql": "mem_available_percent < 100",
"severity": 2
}
]
},
"prom_eval_interval": 15,
"callbacks": [],
"runbook_url": "",
"notify_recovered": 1,
"notify_channels": [
"wecom"
],
"notify_groups": [
"2"
],
"notify_groups_obj": [
{
"id": 2,
"name": "\u6d4b\u8bd5\u90ae\u4ef6\u544a\u8b66\u7684\u56e2\u961f",
"note": "",
"create_at": 1708921626,
"create_by": "root",
"update_at": 1708948109,
"update_by": "root"
}
],
"target_ident": "ulric-flashcat.local",
"target_note": "",
"trigger_time": 1709004236,
"trigger_value": "15.79847",
"trigger_values": "",
"tags": [
"__name__=mem_available_percent",
"ident=ulric-flashcat.local",
"rulename=\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22"
],
"tags_map": {
"__name__": "mem_available_percent",
"ident": "ulric-flashcat.local",
"rulename": "\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22"
},
"annotations": {},
"is_recovered": false,
"notify_users_obj": [
{
"id": 1,
"username": "root",
"nickname": "\u8d85\u7ba1",
"phone": "",
"email": "",
"portrait": "",
"roles": [
"Admin"
],
"contacts": {},
"maintainer": 0,
"create_at": 1708920315,
"create_by": "system",
"update_at": 1708920315,
"update_by": "system",
"admin": true
},
{
"id": 2,
"username": "qinxiaohui",
"nickname": "\u79e6\u6653\u8f89",
"phone": "",
"email": "qinxiaohui@flashcat.cloud",
"portrait": "",
"roles": [
"Standard"
],
"contacts": {},
"maintainer": 0,
"create_at": 1708921503,
"create_by": "root",
"update_at": 1708921503,
"update_by": "root",
"admin": false
},
{
"id": 3,
"username": "n9e-wecom-robot",
"nickname": "\u591c\u83baV7\u7fa4\u673a\u5668\u4eba",
"phone": "",
"email": "",
"portrait": "",
"roles": [
"Guest"
],
"contacts": {
"wecom_robot_token": "https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=x"
},
"maintainer": 0,
"create_at": 1708945529,
"create_by": "root",
"update_at": 1708945529,
"update_by": "root",
"admin": false
},
{
"id": 4,
"username": "n9e-ding-robot",
"nickname": "\u9489\u9489\u673a\u5668\u4eba",
"phone": "",
"email": "",
"portrait": "",
"roles": [
"Guest"
],
"contacts": {
"dingtalk_robot_token": "https://oapi.dingtalk.com/robot/send?access_token=x"
},
"maintainer": 0,
"create_at": 1708948099,
"create_by": "root",
"update_at": 1708948099,
"update_by": "root",
"admin": false
}
],
"last_eval_time": 1709004236,
"last_sent_time": 1709004236,
"notify_cur_number": 1,
"first_trigger_time": 1709004236,
"extra_config": null,
"status": 0,
"claimant": "",
"sub_rule_id": 0,
"extra_info": null
},
"tpls": {
"dingtalk": "#### <font color=\"#FF0000\">S2 - Triggered - \u6d4b\u8bd5\u901a\u77e5\u811a\u672c22</font>\n\n---\n\n- **\u89c4\u5219\u6807\u9898**: \u6d4b\u8bd5\u901a\u77e5\u811a\u672c22\n- **\u89e6\u53d1\u65f6\u503c**: 15.79847\n- **\u76d1\u63a7\u5bf9\u8c61**: ulric-flashcat.local\n- **\u76d1\u63a7\u6307\u6807**: [__name__=mem_available_percent ident=ulric-flashcat.local rulename=\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22]\n- **\u89e6\u53d1\u65f6\u95f4**: 2024-02-27 11:23:56\n- **\u53d1\u9001\u65f6\u95f4**: 2024-02-27 11:23:57\n\t",
"email": "<!DOCTYPE html>\n\t<html lang=\"en\">\n\t<head>\n\t\t<meta charset=\"UTF-8\">\n\t\t<meta http-equiv=\"X-UA-Compatible\" content=\"ie=edge\">\n\t\t<title>\u591c\u83ba\u544a\u8b66\u901a\u77e5</title>\n\t\t<style type=\"text/css\">\n\t\t\t.wrapper {\n\t\t\t\tbackground-color: #f8f8f8;\n\t\t\t\tpadding: 15px;\n\t\t\t\theight: 100%;\n\t\t\t}\n\t\t\t.main {\n\t\t\t\twidth: 600px;\n\t\t\t\tpadding: 30px;\n\t\t\t\tmargin: 0 auto;\n\t\t\t\tbackground-color: #fff;\n\t\t\t\tfont-size: 12px;\n\t\t\t\tfont-family: verdana,'Microsoft YaHei',Consolas,'Deja Vu Sans Mono','Bitstream Vera Sans Mono';\n\t\t\t}\n\t\t\theader {\n\t\t\t\tborder-radius: 2px 2px 0 0;\n\t\t\t}\n\t\t\theader .title {\n\t\t\t\tfont-size: 14px;\n\t\t\t\tcolor: #333333;\n\t\t\t\tmargin: 0;\n\t\t\t}\n\t\t\theader .sub-desc {\n\t\t\t\tcolor: #333;\n\t\t\t\tfont-size: 14px;\n\t\t\t\tmargin-top: 6px;\n\t\t\t\tmargin-bottom: 0;\n\t\t\t}\n\t\t\thr {\n\t\t\t\tmargin: 20px 0;\n\t\t\t\theight: 0;\n\t\t\t\tborder: none;\n\t\t\t\tborder-top: 1px solid #e5e5e5;\n\t\t\t}\n\t\t\tem {\n\t\t\t\tfont-weight: 600;\n\t\t\t}\n\t\t\ttable {\n\t\t\t\tmargin: 20px 0;\n\t\t\t\twidth: 100%;\n\t\t\t}\n\t\n\t\t\ttable tbody tr{\n\t\t\t\tfont-weight: 200;\n\t\t\t\tfont-size: 12px;\n\t\t\t\tcolor: #666;\n\t\t\t\theight: 32px;\n\t\t\t}\n\t\n\t\t\t.succ {\n\t\t\t\tbackground-color: green;\n\t\t\t\tcolor: #fff;\n\t\t\t}\n\t\n\t\t\t.fail {\n\t\t\t\tbackground-color: red;\n\t\t\t\tcolor: #fff;\n\t\t\t}\n\t\n\t\t\t.succ th, .succ td, .fail th, .fail td {\n\t\t\t\tcolor: #fff;\n\t\t\t}\n\t\n\t\t\ttable tbody tr th {\n\t\t\t\twidth: 80px;\n\t\t\t\ttext-align: right;\n\t\t\t}\n\t\t\t.text-right {\n\t\t\t\ttext-align: right;\n\t\t\t}\n\t\t\t.body {\n\t\t\t\tmargin-top: 24px;\n\t\t\t}\n\t\t\t.body-text {\n\t\t\t\tcolor: #666666;\n\t\t\t\t-webkit-font-smoothing: antialiased;\n\t\t\t}\n\t\t\t.body-extra {\n\t\t\t\t-webkit-font-smoothing: antialiased;\n\t\t\t}\n\t\t\t.body-extra.text-right a {\n\t\t\t\ttext-decoration: none;\n\t\t\t\tcolor: #333;\n\t\t\t}\n\t\t\t.body-extra.text-right a:hover {\n\t\t\t\tcolor: #666;\n\t\t\t}\n\t\t\t.button {\n\t\t\t\twidth: 200px;\n\t\t\t\theight: 50px;\n\t\t\t\tmargin-top: 20px;\n\t\t\t\ttext-align: center;\n\t\t\t\tborder-radius: 2px;\n\t\t\t\tbackground: #2D77EE;\n\t\t\t\tline-height: 50px;\n\t\t\t\tfont-size: 20px;\n\t\t\t\tcolor: #FFFFFF;\n\t\t\t\tcursor: pointer;\n\t\t\t}\n\t\t\t.button:hover {\n\t\t\t\tbackground: rgb(25, 115, 255);\n\t\t\t\tborder-color: rgb(25, 115, 255);\n\t\t\t\tcolor: #fff;\n\t\t\t}\n\t\t\tfooter {\n\t\t\t\tmargin-top: 10px;\n\t\t\t\ttext-align: right;\n\t\t\t}\n\t\t\t.footer-logo {\n\t\t\t\ttext-align: right;\n\t\t\t}\n\t\t\t.footer-logo-image {\n\t\t\t\twidth: 108px;\n\t\t\t\theight: 27px;\n\t\t\t\tmargin-right: 10px;\n\t\t\t}\n\t\t\t.copyright {\n\t\t\t\tmargin-top: 10px;\n\t\t\t\tfont-size: 12px;\n\t\t\t\ttext-align: right;\n\t\t\t\tcolor: #999;\n\t\t\t\t-webkit-font-smoothing: antialiased;\n\t\t\t}\n\t\t</style>\n\t</head>\n\t<body>\n\t<div class=\"wrapper\">\n\t\t<div class=\"main\">\n\t\t\t<header>\n\t\t\t\t<h3 class=\"title\">\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22</h3>\n\t\t\t\t<p class=\"sub-desc\"></p>\n\t\t\t</header>\n\t\n\t\t\t<hr>\n\t\n\t\t\t<div class=\"body\">\n\t\t\t\t<table cellspacing=\"0\" cellpadding=\"0\" border=\"0\">\n\t\t\t\t\t<tbody>\n\t\t\t\t\t\n\t\t\t\t\t<tr class=\"fail\">\n\t\t\t\t\t\t<th>\u7ea7\u522b\u72b6\u6001\uff1a</th>\n\t\t\t\t\t\t<td>S2 Triggered</td>\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t<th>\u7b56\u7565\u5907\u6ce8\uff1a</th>\n\t\t\t\t\t\t<td></td>\n\t\t\t\t\t</tr>\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t<th>\u8bbe\u5907\u5907\u6ce8\uff1a</th>\n\t\t\t\t\t\t<td></td>\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t<th>\u89e6\u53d1\u65f6\u503c\uff1a</th>\n\t\t\t\t\t\t<td>15.79847</td>\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\n\t\t\t\t\t\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t<th>\u76d1\u63a7\u5bf9\u8c61\uff1a</th>\n\t\t\t\t\t\t<td>ulric-flashcat.local</td>\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t<th>\u76d1\u63a7\u6307\u6807\uff1a</th>\n\t\t\t\t\t\t<td>[__name__=mem_available_percent ident=ulric-flashcat.local rulename=\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22]</td>\n\t\t\t\t\t</tr>\n\t\n\t\t\t\t\t\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t<th>\u89e6\u53d1\u65f6\u95f4\uff1a</th>\n\t\t\t\t\t\t<td>\n\t\t\t\t\t\t\t2024-02-27 11:23:56\n\t\t\t\t\t\t</td>\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t<th>\u53d1\u9001\u65f6\u95f4\uff1a</th>\n\t\t\t\t\t\t<td>\n\t\t\t\t\t\t\t2024-02-27 11:23:57\n\t\t\t\t\t\t</td>\n\t\t\t\t\t</tr>\n\t\t\t\t\t</tbody>\n\t\t\t\t</table>\n\t\n\t\t\t\t<hr>\n\t\n\t\t\t\t<footer>\n\t\t\t\t\t<div class=\"copyright\" style=\"font-style: italic\">\n\t\t\t\t\t\t\u62a5\u8b66\u592a\u591a\uff1f\u4f7f\u7528 <a href=\"https://flashcat.cloud/product/flashduty/\" target=\"_blank\">FlashDuty</a> \u505a\u544a\u8b66\u805a\u5408\u964d\u566a\u3001\u6392\u73edOnCall\uff01\n\t\t\t\t\t</div>\n\t\t\t\t</footer>\n\t\t\t</div>\n\t\t</div>\n\t</div>\n\t</body>\n\t</html>",
"feishu": "\u7ea7\u522b\u72b6\u6001: S2 Triggered \n\u89c4\u5219\u540d\u79f0: \u6d4b\u8bd5\u901a\u77e5\u811a\u672c22 \n\u76d1\u63a7\u6307\u6807: [__name__=mem_available_percent ident=ulric-flashcat.local rulename=\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22]\n\u89e6\u53d1\u65f6\u95f4: 2024-02-27 11:23:56\n\u89e6\u53d1\u65f6\u503c: 15.79847\n\u53d1\u9001\u65f6\u95f4: 2024-02-27 11:23:57",
"feishucard": " \n**\u544a\u8b66\u96c6\u7fa4:** xxx \n**\u7ea7\u522b\u72b6\u6001:** S2 Triggered \n**\u544a\u8b66\u540d\u79f0:** \u6d4b\u8bd5\u901a\u77e5\u811a\u672c22 \n**\u89e6\u53d1\u65f6\u95f4:** 2024-02-27 11:23:56 \n**\u53d1\u9001\u65f6\u95f4:** 2024-02-27 11:23:57 \n**\u89e6\u53d1\u65f6\u503c:** 15.79847 \n",
"mailsubject": "Triggered: \u6d4b\u8bd5\u901a\u77e5\u811a\u672c22 [__name__=mem_available_percent ident=ulric-flashcat.local rulename=\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22]",
"mm": "\u7ea7\u522b\u72b6\u6001: S2 Triggered \n\u89c4\u5219\u540d\u79f0: \u6d4b\u8bd5\u901a\u77e5\u811a\u672c22 \n\u76d1\u63a7\u6307\u6807: [__name__=mem_available_percent ident=ulric-flashcat.local rulename=\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22] \n\u89e6\u53d1\u65f6\u95f4: 2024-02-27 11:23:56 \n\u89e6\u53d1\u65f6\u503c: 15.79847 \n\u53d1\u9001\u65f6\u95f4: 2024-02-27 11:23:57",
"telegram": "**\u7ea7\u522b\u72b6\u6001**: <font color=\"warning\">S2 Triggered</font> \n**\u89c4\u5219\u6807\u9898**: \u6d4b\u8bd5\u901a\u77e5\u811a\u672c22 \n**\u76d1\u63a7\u5bf9\u8c61**: ulric-flashcat.local \n**\u76d1\u63a7\u6307\u6807**: [__name__=mem_available_percent ident=ulric-flashcat.local rulename=\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22] \n**\u89e6\u53d1\u65f6\u503c**: 15.79847 \n**\u9996\u6b21\u89e6\u53d1\u65f6\u95f4**: 2024-02-27 11:23:56 \n**\u8ddd\u79bb\u9996\u6b21\u544a\u8b66**: 1s\n**\u53d1\u9001\u65f6\u95f4**: 2024-02-27 11:23:57",
"wecom": "**\u7ea7\u522b\u72b6\u6001**: <font color=\"warning\">S2 Triggered</font> \n**\u89c4\u5219\u6807\u9898**: \u6d4b\u8bd5\u901a\u77e5\u811a\u672c22 \n**\u76d1\u63a7\u5bf9\u8c61**: ulric-flashcat.local \n**\u76d1\u63a7\u6307\u6807**: [__name__=mem_available_percent ident=ulric-flashcat.local rulename=\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22] \n**\u89e6\u53d1\u65f6\u503c**: 15.79847 \n**\u9996\u6b21\u89e6\u53d1\u65f6\u95f4**: 2024-02-27 11:23:56 \n**\u8ddd\u79bb\u9996\u6b21\u544a\u8b66**: 1s\n**\u53d1\u9001\u65f6\u95f4**: 2024-02-27 11:23:57"
}
}
补充一个博客,也是写的如何在夜莺里通过自定义脚本发送短信,供大家参考:《夜莺短信告警教程》。