使用通知脚本自定义发送告警消息

上一节介绍的回调推送方式，是一种极其灵活的推送方式，比如 FlashDuty 就是使用这种方式和夜莺对接，但是 Webhook 的方式没法复用夜莺内置的告警模板，这一节介绍另一种极其灵活的推送方式：通知脚本。

夜莺产生事件之后，会调用你指定的通知脚本，把告警事件的详细内容通过 stdin 的方式传给脚本，你就可以在脚本里做各种自定义的逻辑了。何为 stdin，请自行 Google。和 Webhook 方式类似，夜莺也会把告警事件序列化为 JSON 格式，不同的是，JSON 中不但会包含事件原始信息，还会包含事件+通知模板渲染出来的结果文本，这样就无需在脚本里处理这个模板渲染逻辑了。

菜单入口：告警通知 - 通知设置 - 通知脚本，如下所示：

20240227103424

如果你要启用这个功能，需要做如下配置：

  • 启用：打开
  • 超时：配置一个脚本超时时间，比如 10，表示 10s 超时，千万不能设置为 0

然后就是告诉夜莺脚本内容是什么，有两个方式，一个是直接在页面上填写脚本内容，另一个是告诉夜莺脚本路径是什么，夜莺会去这个路径读取脚本内容。如果是脚本路径的方式，需要确保脚本提前放到夜莺进程所在的机器上，而且夜莺进程有读取权限，而且脚本自身要有可执行权限。

这里 放了几个 notify 脚本，可以参考。比如 notify_feishu.py 这个脚本，这是刚开始夜莺还没有内置支持飞书的时候写的，现在夜莺已经在 Go 代码里支持飞书了，所以这个脚本已经不再需要了，但是可以作为参考。其内容如下：

  1. #!/usr/bin/env python
  2. # -*- coding: UTF-8 -*-
  3. import sys
  4. import json
  5. import requests
  7. class Sender(object):
  8.     @classmethod
  9.     def send_email(cls, payload):
  10.         # already done in go code
  11.         pass
  13.     @classmethod
  14.     def send_wecom(cls, payload):
  15.         # already done in go code
  16.         pass
  18.     @classmethod
  19.     def send_dingtalk(cls, payload):
  20.         # already done in go code
  21.         pass
  23.     @classmethod
  24.     def send_ifeishu(cls, payload):
  25.         users = payload.get('event').get("notify_users_obj")
  26.         tokens = {}
  27.         phones = {}
  29.         for u in users:
  30.             if u.get("phone"):
  31.                 phones[u.get("phone")] = 1
  33.             contacts = u.get("contacts")
  34.             if contacts.get("feishu_robot_token", ""):
  35.                 tokens[contacts.get("feishu_robot_token", "")] = 1
  37.         headers = {
  38.             "Content-Type": "application/json;charset=utf-8",
  39.             "Host": "open.feishu.cn"
  40.         }
  42.         for t in tokens:
  43.             url = "https://open.feishu.cn/open-apis/bot/v2/hook/{}".format(t)
  44.             body = {
  45.                 "msg_type": "text",
  46.                 "content": {
  47.                     "text": payload.get('tpls').get("feishu", "feishu not found")
  48.                 },
  49.                 "at": {
  50.                     "atMobiles": list(phones.keys()),
  51.                     "isAtAll": False
  52.                 }
  53.             }
  55.             response = requests.post(url, headers=headers, data=json.dumps(body))
  56.             print(f"notify_ifeishu: token={t} status_code={response.status_code} response_text={response.text}")
  58.     @classmethod
  59.     def send_mm(cls, payload):
  60.         # already done in go code
  61.         pass
  63.     @classmethod
  64.     def send_sms(cls, payload):
  65.         pass
  67.     @classmethod
  68.     def send_voice(cls, payload):
  69.         pass
  71. def main():
  72.     payload = json.load(sys.stdin)
  73.     with open(".payload", 'w') as f:
  74.         f.write(json.dumps(payload, indent=4))
  75.     for ch in payload.get('event').get('notify_channels'):
  76.         send_func_name = "send_{}".format(ch.strip())
  77.         if not hasattr(Sender, send_func_name):
  78.             print("function: {} not found", send_func_name)
  79.             continue
  80.         send_func = getattr(Sender, send_func_name)
  81.         send_func(payload)
  83. def hello():
  84.     print("hello nightingale")
  86. if __name__ == "__main__":
  87.     if len(sys.argv) == 1:
  88.         main()
  89.     elif sys.argv[1] == "hello":
  90.         hello()
  91.     else:
  92.         print("I am confused")

这个脚本用到了 Python requests 库，需要自行安装。脚本核心逻辑在 main 方法中，我们重点看一下这几行代码：

  1. def main():
  2.     payload = json.load(sys.stdin)
  3.     with open(".payload", 'w') as f:
  4.         f.write(json.dumps(payload, indent=4))
  5.     for ch in payload.get('event').get('notify_channels'):
  6.         send_func_name = "send_{}".format(ch.strip())
  7.         if not hasattr(Sender, send_func_name):
  8.             print("function: {} not found", send_func_name)
  9.             continue
  10.         send_func = getattr(Sender, send_func_name)
  11.         send_func(payload)

json.load(sys.stdin) 是从 stdin 中读取 JSON 数据，这个 JSON 数据就是告警事件的详细内容。然后把 stdin 里读取的内容写到了 .payload 文件中了，后面可以从 .payload 文件中看到具体 json 长什么样。

然后遍历 notify_channels，这个字段是告警规则里配置的通知媒介，比如 email、wecom、dingtalk、feishu、mm、sms、voice 等。然后拼接成 send_ 开头的函数名，比如 send_email、send_wecom、send_dingtalk、send_feishu、send_mm、send_sms、send_voice 等。然后用 Python 的反射机制，调用这个函数，把告警事件的详细内容传给这个函数。

然后具体的发送逻辑就在各个 send 函数中实现了，比如 send_feishu 函数中，就是调用飞书的 Webhook 接口，把告警事件的详细内容传给飞书。

实际上，你也可以自定义通知媒介，菜单入口：告警通知 - 通知设置 - 通知媒介，然后配合自定义通知模板、user 的自定义联系方式、以及自定义通知脚本，就可以完成自定义通知逻辑了。可以据此实现电话、短信通知，甚至是自定义的 IM 通知。当然，这整个流程有点复杂，需要你对整个原理非常清楚。还不如直接使用 FlashDuty 做通知，省心一万倍。

最后，我给大家贴一下 .payload 中的内容，这个数据格式也是社区咨询比较多的，样例如下：

  1. {
  2.     "event": {
  3.         "id": 20,
  4.         "cate": "prometheus",
  5.         "cluster": "xxx",
  6.         "datasource_id": 1,
  7.         "group_id": 1,
  8.         "group_name": "Default Busi Group",
  9.         "hash": "1951d1a34a6f1be4697269268a1c257f",
  10.         "rule_id": 6,
  11.         "rule_name": "\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22",
  12.         "rule_note": "",
  13.         "rule_prod": "metric",
  14.         "rule_algo": "",
  15.         "severity": 2,
  16.         "prom_for_duration": 0,
  17.         "prom_ql": "mem_available_percent < 100",
  18.         "rule_config": {
  19.             "queries": [
  20.                 {
  21.                     "keys": {
  22.                         "labelKey": "",
  23.                         "valueKey": ""
  24.                     },
  25.                     "prom_ql": "mem_available_percent < 100",
  26.                     "severity": 2
  27.                 }
  28.             ]
  29.         },
  30.         "prom_eval_interval": 15,
  31.         "callbacks": [],
  32.         "runbook_url": "",
  33.         "notify_recovered": 1,
  34.         "notify_channels": [
  35.             "wecom"
  36.         ],
  37.         "notify_groups": [
  38.             "2"
  39.         ],
  40.         "notify_groups_obj": [
  41.             {
  42.                 "id": 2,
  43.                 "name": "\u6d4b\u8bd5\u90ae\u4ef6\u544a\u8b66\u7684\u56e2\u961f",
  44.                 "note": "",
  45.                 "create_at": 1708921626,
  46.                 "create_by": "root",
  47.                 "update_at": 1708948109,
  48.                 "update_by": "root"
  49.             }
  50.         ],
  51.         "target_ident": "ulric-flashcat.local",
  52.         "target_note": "",
  53.         "trigger_time": 1709004236,
  54.         "trigger_value": "15.79847",
  55.         "trigger_values": "",
  56.         "tags": [
  57.             "__name__=mem_available_percent",
  58.             "ident=ulric-flashcat.local",
  59.             "rulename=\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22"
  60.         ],
  61.         "tags_map": {
  62.             "__name__": "mem_available_percent",
  63.             "ident": "ulric-flashcat.local",
  64.             "rulename": "\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22"
  65.         },
  66.         "annotations": {},
  67.         "is_recovered": false,
  68.         "notify_users_obj": [
  69.             {
  70.                 "id": 1,
  71.                 "username": "root",
  72.                 "nickname": "\u8d85\u7ba1",
  73.                 "phone": "",
  74.                 "email": "",
  75.                 "portrait": "",
  76.                 "roles": [
  77.                     "Admin"
  78.                 ],
  79.                 "contacts": {},
  80.                 "maintainer": 0,
  81.                 "create_at": 1708920315,
  82.                 "create_by": "system",
  83.                 "update_at": 1708920315,
  84.                 "update_by": "system",
  85.                 "admin": true
  86.             },
  87.             {
  88.                 "id": 2,
  89.                 "username": "qinxiaohui",
  90.                 "nickname": "\u79e6\u6653\u8f89",
  91.                 "phone": "",
  92.                 "email": "qinxiaohui@flashcat.cloud",
  93.                 "portrait": "",
  94.                 "roles": [
  95.                     "Standard"
  96.                 ],
  97.                 "contacts": {},
  98.                 "maintainer": 0,
  99.                 "create_at": 1708921503,
  100.                 "create_by": "root",
  101.                 "update_at": 1708921503,
  102.                 "update_by": "root",
  103.                 "admin": false
  104.             },
  105.             {
  106.                 "id": 3,
  107.                 "username": "n9e-wecom-robot",
  108.                 "nickname": "\u591c\u83baV7\u7fa4\u673a\u5668\u4eba",
  109.                 "phone": "",
  110.                 "email": "",
  111.                 "portrait": "",
  112.                 "roles": [
  113.                     "Guest"
  114.                 ],
  115.                 "contacts": {
  116.                     "wecom_robot_token": "https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=x"
  117.                 },
  118.                 "maintainer": 0,
  119.                 "create_at": 1708945529,
  120.                 "create_by": "root",
  121.                 "update_at": 1708945529,
  122.                 "update_by": "root",
  123.                 "admin": false
  124.             },
  125.             {
  126.                 "id": 4,
  127.                 "username": "n9e-ding-robot",
  128.                 "nickname": "\u9489\u9489\u673a\u5668\u4eba",
  129.                 "phone": "",
  130.                 "email": "",
  131.                 "portrait": "",
  132.                 "roles": [
  133.                     "Guest"
  134.                 ],
  135.                 "contacts": {
  136.                     "dingtalk_robot_token": "https://oapi.dingtalk.com/robot/send?access_token=x"
  137.                 },
  138.                 "maintainer": 0,
  139.                 "create_at": 1708948099,
  140.                 "create_by": "root",
  141.                 "update_at": 1708948099,
  142.                 "update_by": "root",
  143.                 "admin": false
  144.             }
  145.         ],
  146.         "last_eval_time": 1709004236,
  147.         "last_sent_time": 1709004236,
  148.         "notify_cur_number": 1,
  149.         "first_trigger_time": 1709004236,
  150.         "extra_config": null,
  151.         "status": 0,
  152.         "claimant": "",
  153.         "sub_rule_id": 0,
  154.         "extra_info": null
  155.     },
  156.     "tpls": {
  157.         "dingtalk": "#### <font color=\"#FF0000\">S2 - Triggered - \u6d4b\u8bd5\u901a\u77e5\u811a\u672c22</font>\n\n---\n\n- **\u89c4\u5219\u6807\u9898**: \u6d4b\u8bd5\u901a\u77e5\u811a\u672c22\n- **\u89e6\u53d1\u65f6\u503c**: 15.79847\n- **\u76d1\u63a7\u5bf9\u8c61**: ulric-flashcat.local\n- **\u76d1\u63a7\u6307\u6807**: [__name__=mem_available_percent ident=ulric-flashcat.local rulename=\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22]\n- **\u89e6\u53d1\u65f6\u95f4**: 2024-02-27 11:23:56\n- **\u53d1\u9001\u65f6\u95f4**: 2024-02-27 11:23:57\n\t",
  158.         "email": "<!DOCTYPE html>\n\t<html lang=\"en\">\n\t<head>\n\t\t<meta charset=\"UTF-8\">\n\t\t<meta http-equiv=\"X-UA-Compatible\" content=\"ie=edge\">\n\t\t<title>\u591c\u83ba\u544a\u8b66\u901a\u77e5</title>\n\t\t<style type=\"text/css\">\n\t\t\t.wrapper {\n\t\t\t\tbackground-color: #f8f8f8;\n\t\t\t\tpadding: 15px;\n\t\t\t\theight: 100%;\n\t\t\t}\n\t\t\t.main {\n\t\t\t\twidth: 600px;\n\t\t\t\tpadding: 30px;\n\t\t\t\tmargin: 0 auto;\n\t\t\t\tbackground-color: #fff;\n\t\t\t\tfont-size: 12px;\n\t\t\t\tfont-family: verdana,'Microsoft YaHei',Consolas,'Deja Vu Sans Mono','Bitstream Vera Sans Mono';\n\t\t\t}\n\t\t\theader {\n\t\t\t\tborder-radius: 2px 2px 0 0;\n\t\t\t}\n\t\t\theader .title {\n\t\t\t\tfont-size: 14px;\n\t\t\t\tcolor: #333333;\n\t\t\t\tmargin: 0;\n\t\t\t}\n\t\t\theader .sub-desc {\n\t\t\t\tcolor: #333;\n\t\t\t\tfont-size: 14px;\n\t\t\t\tmargin-top: 6px;\n\t\t\t\tmargin-bottom: 0;\n\t\t\t}\n\t\t\thr {\n\t\t\t\tmargin: 20px 0;\n\t\t\t\theight: 0;\n\t\t\t\tborder: none;\n\t\t\t\tborder-top: 1px solid #e5e5e5;\n\t\t\t}\n\t\t\tem {\n\t\t\t\tfont-weight: 600;\n\t\t\t}\n\t\t\ttable {\n\t\t\t\tmargin: 20px 0;\n\t\t\t\twidth: 100%;\n\t\t\t}\n\t\n\t\t\ttable tbody tr{\n\t\t\t\tfont-weight: 200;\n\t\t\t\tfont-size: 12px;\n\t\t\t\tcolor: #666;\n\t\t\t\theight: 32px;\n\t\t\t}\n\t\n\t\t\t.succ {\n\t\t\t\tbackground-color: green;\n\t\t\t\tcolor: #fff;\n\t\t\t}\n\t\n\t\t\t.fail {\n\t\t\t\tbackground-color: red;\n\t\t\t\tcolor: #fff;\n\t\t\t}\n\t\n\t\t\t.succ th, .succ td, .fail th, .fail td {\n\t\t\t\tcolor: #fff;\n\t\t\t}\n\t\n\t\t\ttable tbody tr th {\n\t\t\t\twidth: 80px;\n\t\t\t\ttext-align: right;\n\t\t\t}\n\t\t\t.text-right {\n\t\t\t\ttext-align: right;\n\t\t\t}\n\t\t\t.body {\n\t\t\t\tmargin-top: 24px;\n\t\t\t}\n\t\t\t.body-text {\n\t\t\t\tcolor: #666666;\n\t\t\t\t-webkit-font-smoothing: antialiased;\n\t\t\t}\n\t\t\t.body-extra {\n\t\t\t\t-webkit-font-smoothing: antialiased;\n\t\t\t}\n\t\t\t.body-extra.text-right a {\n\t\t\t\ttext-decoration: none;\n\t\t\t\tcolor: #333;\n\t\t\t}\n\t\t\t.body-extra.text-right a:hover {\n\t\t\t\tcolor: #666;\n\t\t\t}\n\t\t\t.button {\n\t\t\t\twidth: 200px;\n\t\t\t\theight: 50px;\n\t\t\t\tmargin-top: 20px;\n\t\t\t\ttext-align: center;\n\t\t\t\tborder-radius: 2px;\n\t\t\t\tbackground: #2D77EE;\n\t\t\t\tline-height: 50px;\n\t\t\t\tfont-size: 20px;\n\t\t\t\tcolor: #FFFFFF;\n\t\t\t\tcursor: pointer;\n\t\t\t}\n\t\t\t.button:hover {\n\t\t\t\tbackground: rgb(25, 115, 255);\n\t\t\t\tborder-color: rgb(25, 115, 255);\n\t\t\t\tcolor: #fff;\n\t\t\t}\n\t\t\tfooter {\n\t\t\t\tmargin-top: 10px;\n\t\t\t\ttext-align: right;\n\t\t\t}\n\t\t\t.footer-logo {\n\t\t\t\ttext-align: right;\n\t\t\t}\n\t\t\t.footer-logo-image {\n\t\t\t\twidth: 108px;\n\t\t\t\theight: 27px;\n\t\t\t\tmargin-right: 10px;\n\t\t\t}\n\t\t\t.copyright {\n\t\t\t\tmargin-top: 10px;\n\t\t\t\tfont-size: 12px;\n\t\t\t\ttext-align: right;\n\t\t\t\tcolor: #999;\n\t\t\t\t-webkit-font-smoothing: antialiased;\n\t\t\t}\n\t\t</style>\n\t</head>\n\t<body>\n\t<div class=\"wrapper\">\n\t\t<div class=\"main\">\n\t\t\t<header>\n\t\t\t\t<h3 class=\"title\">\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22</h3>\n\t\t\t\t<p class=\"sub-desc\"></p>\n\t\t\t</header>\n\t\n\t\t\t<hr>\n\t\n\t\t\t<div class=\"body\">\n\t\t\t\t<table cellspacing=\"0\" cellpadding=\"0\" border=\"0\">\n\t\t\t\t\t<tbody>\n\t\t\t\t\t\n\t\t\t\t\t<tr class=\"fail\">\n\t\t\t\t\t\t<th>\u7ea7\u522b\u72b6\u6001\uff1a</th>\n\t\t\t\t\t\t<td>S2 Triggered</td>\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t<th>\u7b56\u7565\u5907\u6ce8\uff1a</th>\n\t\t\t\t\t\t<td></td>\n\t\t\t\t\t</tr>\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t<th>\u8bbe\u5907\u5907\u6ce8\uff1a</th>\n\t\t\t\t\t\t<td></td>\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t<th>\u89e6\u53d1\u65f6\u503c\uff1a</th>\n\t\t\t\t\t\t<td>15.79847</td>\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\n\t\t\t\t\t\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t<th>\u76d1\u63a7\u5bf9\u8c61\uff1a</th>\n\t\t\t\t\t\t<td>ulric-flashcat.local</td>\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t<th>\u76d1\u63a7\u6307\u6807\uff1a</th>\n\t\t\t\t\t\t<td>[__name__=mem_available_percent ident=ulric-flashcat.local rulename=\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22]</td>\n\t\t\t\t\t</tr>\n\t\n\t\t\t\t\t\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t<th>\u89e6\u53d1\u65f6\u95f4\uff1a</th>\n\t\t\t\t\t\t<td>\n\t\t\t\t\t\t\t2024-02-27 11:23:56\n\t\t\t\t\t\t</td>\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t<th>\u53d1\u9001\u65f6\u95f4\uff1a</th>\n\t\t\t\t\t\t<td>\n\t\t\t\t\t\t\t2024-02-27 11:23:57\n\t\t\t\t\t\t</td>\n\t\t\t\t\t</tr>\n\t\t\t\t\t</tbody>\n\t\t\t\t</table>\n\t\n\t\t\t\t<hr>\n\t\n\t\t\t\t<footer>\n\t\t\t\t\t<div class=\"copyright\" style=\"font-style: italic\">\n\t\t\t\t\t\t\u62a5\u8b66\u592a\u591a\uff1f\u4f7f\u7528 <a href=\"https://flashcat.cloud/product/flashduty/\" target=\"_blank\">FlashDuty</a> \u505a\u544a\u8b66\u805a\u5408\u964d\u566a\u3001\u6392\u73edOnCall\uff01\n\t\t\t\t\t</div>\n\t\t\t\t</footer>\n\t\t\t</div>\n\t\t</div>\n\t</div>\n\t</body>\n\t</html>",
  159.         "feishu": "\u7ea7\u522b\u72b6\u6001: S2 Triggered   \n\u89c4\u5219\u540d\u79f0: \u6d4b\u8bd5\u901a\u77e5\u811a\u672c22   \n\u76d1\u63a7\u6307\u6807: [__name__=mem_available_percent ident=ulric-flashcat.local rulename=\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22]\n\u89e6\u53d1\u65f6\u95f4: 2024-02-27 11:23:56\n\u89e6\u53d1\u65f6\u503c: 15.79847\n\u53d1\u9001\u65f6\u95f4: 2024-02-27 11:23:57",
  160.         "feishucard": "   \n**\u544a\u8b66\u96c6\u7fa4:** xxx   \n**\u7ea7\u522b\u72b6\u6001:** S2 Triggered   \n**\u544a\u8b66\u540d\u79f0:** \u6d4b\u8bd5\u901a\u77e5\u811a\u672c22   \n**\u89e6\u53d1\u65f6\u95f4:** 2024-02-27 11:23:56   \n**\u53d1\u9001\u65f6\u95f4:** 2024-02-27 11:23:57   \n**\u89e6\u53d1\u65f6\u503c:** 15.79847   \n",
  161.         "mailsubject": "Triggered: \u6d4b\u8bd5\u901a\u77e5\u811a\u672c22 [__name__=mem_available_percent ident=ulric-flashcat.local rulename=\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22]",
  162.         "mm": "\u7ea7\u522b\u72b6\u6001: S2 Triggered   \n\u89c4\u5219\u540d\u79f0: \u6d4b\u8bd5\u901a\u77e5\u811a\u672c22   \n\u76d1\u63a7\u6307\u6807: [__name__=mem_available_percent ident=ulric-flashcat.local rulename=\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22]   \n\u89e6\u53d1\u65f6\u95f4: 2024-02-27 11:23:56   \n\u89e6\u53d1\u65f6\u503c: 15.79847   \n\u53d1\u9001\u65f6\u95f4: 2024-02-27 11:23:57",
  163.         "telegram": "**\u7ea7\u522b\u72b6\u6001**: <font color=\"warning\">S2 Triggered</font>   \n**\u89c4\u5219\u6807\u9898**: \u6d4b\u8bd5\u901a\u77e5\u811a\u672c22   \n**\u76d1\u63a7\u5bf9\u8c61**: ulric-flashcat.local   \n**\u76d1\u63a7\u6307\u6807**: [__name__=mem_available_percent ident=ulric-flashcat.local rulename=\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22]   \n**\u89e6\u53d1\u65f6\u503c**: 15.79847   \n**\u9996\u6b21\u89e6\u53d1\u65f6\u95f4**: 2024-02-27 11:23:56   \n**\u8ddd\u79bb\u9996\u6b21\u544a\u8b66**: 1s\n**\u53d1\u9001\u65f6\u95f4**: 2024-02-27 11:23:57",
  164.         "wecom": "**\u7ea7\u522b\u72b6\u6001**: <font color=\"warning\">S2 Triggered</font>   \n**\u89c4\u5219\u6807\u9898**: \u6d4b\u8bd5\u901a\u77e5\u811a\u672c22   \n**\u76d1\u63a7\u5bf9\u8c61**: ulric-flashcat.local   \n**\u76d1\u63a7\u6307\u6807**: [__name__=mem_available_percent ident=ulric-flashcat.local rulename=\u6d4b\u8bd5\u901a\u77e5\u811a\u672c22]   \n**\u89e6\u53d1\u65f6\u503c**: 15.79847   \n**\u9996\u6b21\u89e6\u53d1\u65f6\u95f4**: 2024-02-27 11:23:56   \n**\u8ddd\u79bb\u9996\u6b21\u544a\u8b66**: 1s\n**\u53d1\u9001\u65f6\u95f4**: 2024-02-27 11:23:57"
  165.     }
  166. }