高可用方案

本文介绍 Zadig 高可用保障相关的配置和建议。

高可用保障

包括 K8s 基础设施、Zadig 系统组件和数据库的高可用。

K8s 资源高可用

基础资源的高可用,由 Kubernetes 集群的提供者(公有云/自建云)保障。

应用高可用

提供以下建议保障应用高可用:

  • 使用 K8s 原生的 deployment 部署,保证服务出现极端情况异常退出后可以被迅速拉起
  • 通过内置的 metric 接口,结合 Prometheus 以及预配置的 Grafana 面板实时展现 Zadig 组件的性能指标(CPU / Memory 利用率)
  • 根据当前环境中实际的业务规模,为各组件配置合适的资源规格。请参考 Zadig Chart 的 values.yaml 文件中 microservice.{service}.resources 配置项进行配置
  • 结合 Prometheus alertmanager 配置核心组件监控报警,报警触发后请及时检查组件的运行状态,部分核心组件的报警规则建议如下:
    • aslan,cron,hub-server 组件:当 CPU 或内存占用率达到资源配置上限的 70%,持续超过 2 分钟时触发报警
    • dind 组件:当 CPU 或内存占用率达到资源配置上限的 80%,持续超过 2 分钟时触发报警
  • 备份当前安装参数以及 Chart:
    • 通过 helm get values <Zadig Release Name> -n <Zadig Namespace> > Zadig.yaml 获取当前安装参数
    • 通过 helm pull koderover-chart/zadig --version=<version> 备份当前 Chart

提示

Zadig 中部分组件支持多副本策略,可根据组件的实时资源占用情况按需调整实际副本数。支持多副本配置的组件如下:

  • deployment/zadig-portal
  • deployment/aslan
  • deployment/user
  • deployment/gateway-proxy
  • statefulSet/dind

数据高可用

建议安装时使用高可用 MongoDB、MySQL 存储介质,并对 Zadig 系统数据定时备份。建议如下:

  • 使用高可用的数据库,以阿里云为例:
    • MySQL 建议使用 RDS MySQL 高可用版
    • MongoDB 选用独占型实例
  • 使用高可用的制品仓库:
    • 使用云厂商提供的高可用镜像仓库 / Helm 仓库
    • 若使用 K8s Helm Chart 项目,建议集成高可用的对象存储并设为默认使用,数据备份建议:一天一次
  • 定期备份数据库数据
    • 若使用公有云,需要手动设置公有云 MySQL/MongoDB 实例的自动备份策略
    • 备份策略建议:一天一次
    • MinIO 备份策略:一天一次

监控告警配置

推荐使用 Prometheus高可用方案 - 图1 (opens new window) 作为监控和性能指标信息存储方案,使用 Grafana高可用方案 - 图2 (opens new window) 作为可视化组件进行展示。

Prometheus 配置

  1. 在 Prometheus 配置的 scrape_configs 中添加如下 job:

如果 Zadig 和 Prometheus 在同一集群中

  1. job_name: prometheus
  2. metrics_path: /api/metrics
  3. static_configs:
  4. - targets:
  5. - aslan.<部署 namespace>.svc:25000

如果 Zadig 和 Prometheus 在不同集群中

  1. job_name: admin
  2. metrics_path: /api/aslan/metrics
  3. static_configs:
  4. - targets:
  5. - <Zadig的访问域名>
  6. scheme: https
  1. 重新加载 Prometheus 配置
  2. 在 Prometheus 的 query 界面,输入 request_total 进行查询,确认有数据,证明配置成功

Grafana 配置

  1. [可选项]在 Grafana 的 configuration - Data source 中配置上文的 Prometheus 为数据源
  2. 在 dashboards - import 中输入如下 JSON 导入 Grafana 面板
  1. {
  2. "annotations": {
  3. "list": [
  4. {
  5. "builtIn": 1,
  6. "datasource": {
  7. "type": "grafana",
  8. "uid": "-- Grafana --"
  9. },
  10. "enable": true,
  11. "hide": true,
  12. "iconColor": "rgba(0, 211, 255, 1)",
  13. "name": "Annotations & Alerts",
  14. "target": {
  15. "limit": 100,
  16. "matchAny": false,
  17. "tags": [],
  18. "type": "dashboard"
  19. },
  20. "type": "dashboard"
  21. }
  22. ]
  23. },
  24. "editable": true,
  25. "fiscalYearStartMonth": 0,
  26. "graphTooltip": 0,
  27. "id": 1,
  28. "links": [],
  29. "liveNow": false,
  30. "panels": [
  31. {
  32. "fieldConfig": {
  33. "defaults": {
  34. "color": {
  35. "mode": "palette-classic"
  36. },
  37. "custom": {
  38. "axisCenteredZero": false,
  39. "axisColorMode": "text",
  40. "axisLabel": "",
  41. "axisPlacement": "auto",
  42. "barAlignment": 0,
  43. "drawStyle": "line",
  44. "fillOpacity": 0,
  45. "gradientMode": "none",
  46. "hideFrom": {
  47. "legend": false,
  48. "tooltip": false,
  49. "viz": false
  50. },
  51. "lineInterpolation": "linear",
  52. "lineWidth": 3,
  53. "pointSize": 5,
  54. "scaleDistribution": {
  55. "type": "linear"
  56. },
  57. "showPoints": "auto",
  58. "spanNulls": false,
  59. "stacking": {
  60. "group": "A",
  61. "mode": "none"
  62. },
  63. "thresholdsStyle": {
  64. "mode": "off"
  65. }
  66. },
  67. "mappings": [],
  68. "thresholds": {
  69. "mode": "absolute",
  70. "steps": [
  71. {
  72. "color": "green",
  73. "value": null
  74. },
  75. {
  76. "color": "red",
  77. "value": 80
  78. }
  79. ]
  80. }
  81. },
  82. "overrides": []
  83. },
  84. "gridPos": {
  85. "h": 9,
  86. "w": 12,
  87. "x": 0,
  88. "y": 0
  89. },
  90. "id": 2,
  91. "options": {
  92. "legend": {
  93. "calcs": [],
  94. "displayMode": "list",
  95. "placement": "right",
  96. "showLegend": true
  97. },
  98. "tooltip": {
  99. "mode": "single",
  100. "sort": "none"
  101. }
  102. },
  103. "targets": [
  104. {
  105. "editorMode": "code",
  106. "exemplar": false,
  107. "expr": "sum by(instance) (rate(request_total{status=\"200\"}[1m]))",
  108. "format": "time_series",
  109. "hide": false,
  110. "instant": false,
  111. "interval": "",
  112. "legendFormat": "200",
  113. "range": true,
  114. "refId": "A"
  115. },
  116. {
  117. "editorMode": "code",
  118. "expr": "sum by(instance) (rate(request_total{status=~\"4..\"}[1m]))",
  119. "hide": false,
  120. "legendFormat": "4xx",
  121. "range": true,
  122. "refId": "B"
  123. },
  124. {
  125. "editorMode": "code",
  126. "expr": "sum by(instance) (rate(request_total{status=\"5..\"}[1m]))",
  127. "hide": false,
  128. "legendFormat": "5xx",
  129. "range": true,
  130. "refId": "C"
  131. }
  132. ],
  133. "title": "QPS追踪",
  134. "type": "timeseries"
  135. },
  136. {
  137. "description": "",
  138. "fieldConfig": {
  139. "defaults": {
  140. "color": {
  141. "mode": "thresholds"
  142. },
  143. "mappings": [],
  144. "thresholds": {
  145. "mode": "absolute",
  146. "steps": [
  147. {
  148. "color": "green",
  149. "value": null
  150. },
  151. {
  152. "color": "red",
  153. "value": 80
  154. }
  155. ]
  156. }
  157. },
  158. "overrides": []
  159. },
  160. "gridPos": {
  161. "h": 9,
  162. "w": 6,
  163. "x": 12,
  164. "y": 0
  165. },
  166. "id": 4,
  167. "options": {
  168. "orientation": "auto",
  169. "reduceOptions": {
  170. "calcs": [
  171. "lastNotNull"
  172. ],
  173. "fields": "",
  174. "values": false
  175. },
  176. "showThresholdLabels": false,
  177. "showThresholdMarkers": true
  178. },
  179. "pluginVersion": "9.4.7",
  180. "targets": [
  181. {
  182. "editorMode": "code",
  183. "expr": "running_workflows",
  184. "legendFormat": "__auto",
  185. "range": true,
  186. "refId": "A"
  187. }
  188. ],
  189. "title": "运行中的工作流",
  190. "type": "gauge"
  191. },
  192. {
  193. "description": "",
  194. "fieldConfig": {
  195. "defaults": {
  196. "color": {
  197. "mode": "thresholds"
  198. },
  199. "mappings": [],
  200. "thresholds": {
  201. "mode": "absolute",
  202. "steps": [
  203. {
  204. "color": "green",
  205. "value": null
  206. },
  207. {
  208. "color": "red",
  209. "value": 80
  210. }
  211. ]
  212. }
  213. },
  214. "overrides": []
  215. },
  216. "gridPos": {
  217. "h": 9,
  218. "w": 6,
  219. "x": 18,
  220. "y": 0
  221. },
  222. "id": 6,
  223. "options": {
  224. "orientation": "auto",
  225. "reduceOptions": {
  226. "calcs": [
  227. "lastNotNull"
  228. ],
  229. "fields": "",
  230. "values": false
  231. },
  232. "showThresholdLabels": false,
  233. "showThresholdMarkers": true
  234. },
  235. "pluginVersion": "9.4.7",
  236. "targets": [
  237. {
  238. "editorMode": "code",
  239. "expr": "pending_workflows",
  240. "legendFormat": "__auto",
  241. "range": true,
  242. "refId": "A"
  243. }
  244. ],
  245. "title": "排队中的工作流",
  246. "type": "gauge"
  247. },
  248. {
  249. "fieldConfig": {
  250. "defaults": {
  251. "color": {
  252. "mode": "palette-classic"
  253. },
  254. "custom": {
  255. "axisCenteredZero": false,
  256. "axisColorMode": "text",
  257. "axisLabel": "",
  258. "axisPlacement": "auto",
  259. "barAlignment": 0,
  260. "drawStyle": "line",
  261. "fillOpacity": 0,
  262. "gradientMode": "none",
  263. "hideFrom": {
  264. "legend": false,
  265. "tooltip": false,
  266. "viz": false
  267. },
  268. "lineInterpolation": "linear",
  269. "lineWidth": 1,
  270. "pointSize": 5,
  271. "scaleDistribution": {
  272. "type": "linear"
  273. },
  274. "showPoints": "auto",
  275. "spanNulls": false,
  276. "stacking": {
  277. "group": "A",
  278. "mode": "none"
  279. },
  280. "thresholdsStyle": {
  281. "mode": "off"
  282. }
  283. },
  284. "mappings": [],
  285. "thresholds": {
  286. "mode": "absolute",
  287. "steps": [
  288. {
  289. "color": "green",
  290. "value": null
  291. },
  292. {
  293. "color": "red",
  294. "value": 80
  295. }
  296. ]
  297. }
  298. },
  299. "overrides": []
  300. },
  301. "gridPos": {
  302. "h": 8,
  303. "w": 12,
  304. "x": 0,
  305. "y": 9
  306. },
  307. "id": 8,
  308. "options": {
  309. "legend": {
  310. "calcs": [],
  311. "displayMode": "list",
  312. "placement": "right",
  313. "showLegend": true
  314. },
  315. "tooltip": {
  316. "mode": "single",
  317. "sort": "none"
  318. }
  319. },
  320. "targets": [
  321. {
  322. "editorMode": "code",
  323. "expr": "cpu",
  324. "legendFormat": "{{service}}",
  325. "range": true,
  326. "refId": "A"
  327. }
  328. ],
  329. "title": "CPU 消耗",
  330. "type": "timeseries"
  331. },
  332. {
  333. "fieldConfig": {
  334. "defaults": {
  335. "color": {
  336. "mode": "palette-classic"
  337. },
  338. "custom": {
  339. "axisCenteredZero": false,
  340. "axisColorMode": "text",
  341. "axisLabel": "",
  342. "axisPlacement": "auto",
  343. "barAlignment": 0,
  344. "drawStyle": "line",
  345. "fillOpacity": 0,
  346. "gradientMode": "none",
  347. "hideFrom": {
  348. "legend": false,
  349. "tooltip": false,
  350. "viz": false
  351. },
  352. "lineInterpolation": "linear",
  353. "lineWidth": 1,
  354. "pointSize": 5,
  355. "scaleDistribution": {
  356. "type": "linear"
  357. },
  358. "showPoints": "auto",
  359. "spanNulls": false,
  360. "stacking": {
  361. "group": "A",
  362. "mode": "none"
  363. },
  364. "thresholdsStyle": {
  365. "mode": "off"
  366. }
  367. },
  368. "mappings": [],
  369. "thresholds": {
  370. "mode": "absolute",
  371. "steps": [
  372. {
  373. "color": "green",
  374. "value": null
  375. },
  376. {
  377. "color": "red",
  378. "value": 80
  379. }
  380. ]
  381. }
  382. },
  383. "overrides": []
  384. },
  385. "gridPos": {
  386. "h": 8,
  387. "w": 12,
  388. "x": 12,
  389. "y": 9
  390. },
  391. "id": 10,
  392. "options": {
  393. "legend": {
  394. "calcs": [],
  395. "displayMode": "list",
  396. "placement": "right",
  397. "showLegend": true
  398. },
  399. "tooltip": {
  400. "mode": "single",
  401. "sort": "none"
  402. }
  403. },
  404. "targets": [
  405. {
  406. "editorMode": "code",
  407. "expr": "memory",
  408. "legendFormat": "{{service}}",
  409. "range": true,
  410. "refId": "A"
  411. }
  412. ],
  413. "title": "内存消耗(MB)",
  414. "type": "timeseries"
  415. },
  416. {
  417. "fieldConfig": {
  418. "defaults": {
  419. "color": {
  420. "mode": "thresholds"
  421. },
  422. "custom": {
  423. "align": "auto",
  424. "cellOptions": {
  425. "type": "auto"
  426. },
  427. "inspect": false
  428. },
  429. "mappings": [],
  430. "thresholds": {
  431. "mode": "absolute",
  432. "steps": [
  433. {
  434. "color": "green",
  435. "value": null
  436. },
  437. {
  438. "color": "red",
  439. "value": 80
  440. }
  441. ]
  442. }
  443. },
  444. "overrides": [
  445. {
  446. "matcher": {
  447. "id": "byName",
  448. "options": "慢接口"
  449. },
  450. "properties": [
  451. {
  452. "id": "custom.width",
  453. "value": 1301
  454. }
  455. ]
  456. }
  457. ]
  458. },
  459. "gridPos": {
  460. "h": 8,
  461. "w": 12,
  462. "x": 0,
  463. "y": 17
  464. },
  465. "id": 12,
  466. "options": {
  467. "footer": {
  468. "countRows": false,
  469. "fields": "",
  470. "reducer": [
  471. "sum"
  472. ],
  473. "show": false
  474. },
  475. "frameIndex": 0,
  476. "showHeader": true,
  477. "sortBy": []
  478. },
  479. "pluginVersion": "9.4.7",
  480. "targets": [
  481. {
  482. "editorMode": "code",
  483. "exemplar": false,
  484. "expr": "topk(10, sum by(method, handler) (api_response_time_bucket{status=\"200\",le=\"+Inf\"}) - sum by(method, handler) (api_response_time_bucket{status=\"200\",le=\"1.4\"}))",
  485. "format": "time_series",
  486. "hide": false,
  487. "legendFormat": "__auto",
  488. "range": true,
  489. "refId": "A"
  490. }
  491. ],
  492. "title": "慢接口列表",
  493. "transformations": [
  494. {
  495. "id": "reduce",
  496. "options": {
  497. "includeTimeField": false,
  498. "labelsToFields": false,
  499. "mode": "seriesToRows",
  500. "reducers": [
  501. "last"
  502. ]
  503. }
  504. },
  505. {
  506. "id": "sortBy",
  507. "options": {
  508. "fields": {},
  509. "sort": [
  510. {
  511. "desc": true,
  512. "field": "Last"
  513. }
  514. ]
  515. }
  516. },
  517. {
  518. "id": "organize",
  519. "options": {
  520. "excludeByName": {},
  521. "indexByName": {},
  522. "renameByName": {
  523. "Field": "慢接口",
  524. "Last": "请求数量"
  525. }
  526. }
  527. }
  528. ],
  529. "type": "table"
  530. }
  531. ],
  532. "refresh": "5s",
  533. "revision": 1,
  534. "schemaVersion": 38,
  535. "style": "dark",
  536. "tags": [],
  537. "templating": {
  538. "list": []
  539. },
  540. "time": {
  541. "from": "now-5m",
  542. "to": "now"
  543. },
  544. "timepicker": {},
  545. "timezone": "",
  546. "title": "Zadig 监控面板",
  547. "uid": "w1U8k1xVk",
  548. "version": 2,
  549. "weekStart": ""
  550. }
  1. 在面板列表中找到 Zadig 监控面板, 确认数据正常展示。

FAQs

参考文档:部署运维 FAQs