Business Log Alarm

In addition to the alarm of Loggie itself, the monitoring alarm of the business log itself is also a common function. For example, if ERROR log is included in the log, an alarm can be sent. This kind of alarm will be closer to the business itself. It is a very good supplement.

Usage

There are two ways to choose:

  • Alarm integrated in collection: Loggie can detect abnormal logs when Agent collects logs, or when Aggregator forward logs, and then sends an alarm.
  • Independent Alarm: Deploy Loggie separately, use Elasticsearch source or other sources to query logs, and then send alarms for detected logs.

Alarm integrated in collection

Principle

Loggie does not need to be independently deployed. However, the matching during collecting will theoretically have a certain impact on the transmission performance, but it is convenient and simple.

logAlert interceptoris used to detect abnormal logs during log processing. The abnormal logs will be encapsulated as alarm events and sent to logAlert topic, and consumed by logAlert listener. logAlert listener supports sending to Prometheus AlertManager currently. If you need to support other alarm channels, please submit Issues or PR.

Configuration

Add logAlert listener:

Config

  1. loggie:
  2.   monitor:
  3.     logger:
  4.       period: 30s
  5.       enabled: true
  6.     listeners:
  7.       logAlert:
  8.         alertManagerAddress: ["http://127.0.0.1:9093"]
  9.         bufferSize: 100
  10.         batchTimeout: 10s
  11.         batchSize: 10
  12.       filesource: ~
  13.       filewatcher: ~
  14.       reload: ~
  15.       queue: ~
  16.       sink: ~
  17.   http:
  18.     enabled: true
  19.     port: 9196

Add logAlert interceptor, and reference it in ClusterLogConfig/LogConfig:

Config

  1. apiVersion: loggie.io/v1beta1
  2. kind: Interceptor
  3. metadata:
  4.   name: logalert
  5. spec:
  6.   interceptors: |
  7.     - type: logAlert
  8.       matcher:
  9.         contains: ["err"]

The alertManager’s webhook can be configured for other services to receive alerts.

Config

  1. receivers:
  2. - name: webhook
  3.   webhook_configs:
  4.     - url: http://127.0.0.1:8787/webhook
  5.       send_resolved: true

When successful, we can view similar logs in the alertManager:

  1. ts=2021-12-22T13:33:08.639Z caller=log.go:124 level=debug component=dispatcher msg="Received alert" alert=[6b723d0][active]
  2. ts=2021-12-22T13:33:38.640Z caller=log.go:124 level=debug component=dispatcher aggrGroup={}:{} msg=flushing alerts=[[6b723d0][active]]
  3. ts=2021-12-22T13:33:38.642Z caller=log.go:124 level=debug component=dispatcher receiver=webhook integration=webhook[0] msg="Notify success" attempts=1

At the same time, the webhook receives a similar alarm:

Example

  1. {
  2. "receiver": "webhook",
  3. "status": "firing",
  4. "alerts": [
  5.     {
  6.         "status": "firing",
  7.         "labels": {
  8.             "host": "fuyideMacBook-Pro.local",
  9.             "source": "a"
  10.         },
  11.         "annotations": {
  12.             "message": "10.244.0.1 - - [13/Dec/2021:12:40:48 +0000] error \"GET / HTTP/1.1\" 404 683",
  13.             "reason": "contained error"
  14.         },
  15.         "startsAt": "2021-12-22T21:33:08.638086+08:00",
  16.         "endsAt": "0001-01-01T00:00:00Z",
  17.         "generatorURL": "",
  18.         "fingerprint": "6b723d0e395b14dc"
  19.     }
  20. ],
  21. "groupLabels": {},
  22. "commonLabels": {
  23.     "host": "node1",
  24.     "source": "a"
  25. },
  26. "commonAnnotations": {
  27.     "message": "10.244.0.1 - - [13/Dec/2021:12:40:48 +0000] error \"GET / HTTP/1.1\" 404 683",
  28.     "reason": "contained error"
  29. },
  30. "externalURL": "http://xxxxxx:9093",
  31. "version": "4",
  32. "groupKey": "{}:{}",
  33. "truncatedAlerts": 0
  34. }

Independent Alarm

Info

Coming soon, stay tuned…