Promtail

Promtail is an agent which ships the contents of local logs to a private Lokiinstance or Grafana Cloud. It is usuallydeployed to every machine that has applications needed to be monitored.

It primarily:

  1. Discovers targets
  2. Attaches labels to log streams
  3. Pushes them to the Loki instance.

Currently, Promtail can tail logs from two sources: local log files and thesystemd journal (on AMD64 machines only).

Log File Discovery

Before Promtail can ship any data from log files to Loki, it needs to find outinformation about its environment. Specifically, this means discoveringapplications emitting log lines to files that need to be monitored.

Promtail borrows the sameservice discovery mechanism from Prometheus,although it currently only supports static and kubernetes servicediscovery. This limitation is due to the fact that promtail is deployed as adaemon to every local machine and, as such, does not discover label from othermachines. kubernetes service discovery fetches required labels from theKubernetes API server while static usually covers all other use cases.

Just like Prometheus, promtail is configured using a scrape_configs stanza.relabel_configs allows for fine-grained control of what to ingest, what todrop, and the final metadata to attach to the log line. Refer to the docs forconfiguring Promtail for more details.

Labeling and Parsing

During service discovery, metadata is determined (pod name, filename, etc.) thatmay be attached to the log line as a label for easier identification whenquerying logs in Loki. Through relabel_configs, discovered labels can bemutated into the desired form.

To allow more sophisticated filtering afterwards, Promtail allows to set labelsnot only from service discovery, but also based on the contents of each logline. The pipeline_stages can be used to add or update labels, correct thetimestamp, or re-write log lines entirely. Refer to the documentation forpipelines for more details.

Shipping

Once Promtail has a set of targets (i.e., things to read from, like files) andall labels are set correctly, it will start tailing (continuously reading) thelogs from targets. Once enough data is read into memory or after a configurabletimeout, it is flushed as a single batch to Loki.

As Promtail reads data from sources (files and systemd journal, if configured),it will track the last offset it read in a positions file. By default, thepositions file is stored at /var/log/positions.yaml. The positions file helpsPromtail continue reading from where it left off in the case of the Promtailinstance restarting.

API

Promtail features an embedded web server exposing a web console at / and the following API endpoints:

GET /ready

This endpoint returns 200 when Promtail is up and running, and there’s at least one working target.

GET /metrics

This endpoint returns Promtail metrics for Prometheus. See“Operations > Observability“ to get a listof exported metrics.

Promtail web server config

The web server exposed by Promtail can be configured in the Promtail .yaml config file:

  1. server:
  2.   http_listen_host: 127.0.0.1
  3.   http_listen_port: 9080