Loki Canary
Loki Canary is a standalone app that audits the log capturing performance ofLoki.
How it works
Loki Canary writes a log to a file and stores the timestamp in an internalarray. The contents look something like this:
1557935669096040040 ppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp
The relevant part of the log entry is the timestamp; the p
s are just fillerbytes to make the size of the log configurable.
An agent (like Promtail) should be configured to read the log file and ship itto Loki.
Meanwhile, Loki Canary will open a WebSocket connection to Loki and will tailthe logs it creates. When a log is received on the WebSocket, the timestampin the log message is compared to the internal array.
If the received log is:
- The next in the array to be received, it is removed from the array and the(current time - log timestamp) is recorded in the
response_latency
histogram. This is the expected behavior for well behaving logs. - Not the next in the array to be received, it is removed from the array, theresponse time is recorded in the
response_latency
histogram, and theout_of_order_entries
counter is incremented. - Not in the array at all, it is checked against a separate list of receivedlogs to either increment the
duplicate_entries
counter or theunexpected_entries
counter.
In the background, Loki Canary also runs a timer which iterates through all ofthe entries in the internal array. If any of the entries are older than theduration specified by the -wait
flag (defaulting to 60s), they are removedfrom the array and the websocket_missing_entries
counter is incremented. Anadditional query is then made directly to Loki for any missing entries todetermine if they are truly missing or only missing from the WebSocket. Ifmissing entries are not found in the direct query, the missing_entries
counteris incremented.
Installation
Binary
Loki Canary is provided as a pre-compiled binary as part of theLoki Releases on GitHub.
Docker
Loki Canary is also provided as a Docker container image:
# change tag to the most recent release
$ docker pull grafana/loki-canary:v0.2.0
Kubernetes
To run on Kubernetes, you can do something simple like:
kubectl run loki-canary --generator=run-pod/v1
--image=grafana/loki-canary:latest --restart=Never --image-pull-policy=IfNotPresent
--labels=name=loki-canary -- -addr=loki:3100
Or you can do something more complex like deploy it as a DaemonSet, there is aTanka setup for this in the production
folder, you can import it usingjsonnet-bundler
:
jb install github.com/grafana/loki-canary/production/ksonnet/loki-canary
Then in your Tanka environment’s main.jsonnet
you’ll want something likethis:
local loki_canary = import 'loki-canary/loki-canary.libsonnet';
loki_canary {
loki_canary_args+:: {
addr: "loki:3100",
port: 80,
labelname: "instance",
interval: "100ms",
size: 1024,
wait: "3m",
},
_config+:: {
namespace: "default",
}
}
Examples
Standalone Pod Implementation of loki-canary
---
apiVersion: v1
kind: Pod
metadata:
labels:
app: loki-canary
name: loki-canary
name: loki-canary
spec:
containers:
- args:
- -addr=loki:3100
image: grafana/loki-canary:latest
imagePullPolicy: IfNotPresent
name: loki-canary
resources: {}
---
apiVersion: v1
kind: Service
metadata:
name: loki-canary
labels:
app: loki-canary
spec:
type: ClusterIP
selector:
app: loki-canary
ports:
- name: metrics
protocol: TCP
port: 3500
targetPort: 3500
DeamonSet Implementation of loki-canary
---
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
labels:
app: loki-canary
name: loki-canary
name: loki-canary
spec:
template:
metadata:
name: loki-canary
labels:
app: loki-canary
spec:
containers:
- args:
- -addr=loki:3100
image: grafana/loki-canary:latest
imagePullPolicy: IfNotPresent
name: loki-canary
resources: {}
---
apiVersion: v1
kind: Service
metadata:
name: loki-canary
labels:
app: loki-canary
spec:
type: ClusterIP
selector:
app: loki-canary
ports:
- name: metrics
protocol: TCP
port: 3500
targetPort: 3500
From Source
If the other options are not sufficient for your use case, you can compileloki-canary
yourself:
# clone the source tree
$ git clone https://github.com/grafana/loki
# build the binary
$ make loki-canary
# (optionally build the container image)
$ make loki-canary-image
Configuration
The address of Loki must be passed in with the -addr
flag, and if your Lokiserver uses TLS, -tls=true
must also be provided. Note that using TLS willcause the WebSocket connection to use wss://
instead of ws://
.
The -labelname
and -labelvalue
flags should also be provided, as these areused by Loki Canary to filter the log stream to only process logs for thecurrent instance of the canary. Ensure that the values provided to the flags areunique to each instance of Loki Canary. Grafana Labs’ Tanka configaccomplishes this by passing in the pod name as the label value.
If Loki Canary reports a high number of unexpected_entries
, Loki Canary maynot be waiting long enough and the value for the -wait
flag should beincreased to a larger value than 60s.
Be aware of the relationship between pruneinterval
and the interval
.For example, with an interval of 10ms (100 logs per second) and a prune intervalof 60s, you will write 6000 logs per minute. If those logs were not receivedover the WebSocket, the canary will attempt to query Loki directly to see ifthey are completely lost. However the query return is limited to 1000results so you will not be able to return all the logs even if they did make itto Loki.
Likewise, if you lower the pruneinterval
you risk causing a denial ofservice attack as all your canaries attempt to query for missing logs atwhatever your pruneinterval
is defined at.
All options:
-addr string
The Loki server URL:Port, e.g. loki:3100
-buckets int
Number of buckets in the response_latency histogram (default 10)
-interval duration
Duration between log entries (default 1s)
-labelname string
The label name for this instance of loki-canary to use in the log selector (default "name")
-labelvalue string
The unique label value for this instance of loki-canary to use in the log selector (default "loki-canary")
-pass string
Loki password
-port int
Port which loki-canary should expose metrics (default 3500)
-pruneinterval duration
Frequency to check sent vs received logs, also the frequency which queries for missing logs will be dispatched to loki (default 1m0s)
-size int
Size in bytes of each log line (default 100)
-tls
Does the loki connection use TLS?
-user string
Loki username
-wait duration
Duration to wait for log entries before reporting them lost (default 1m0s)