System Logs

System component logs record events happening in cluster, which can be very useful for debugging. You can configure log verbosity to see more or less detail. Logs can be as coarse-grained as showing errors within a component, or as fine-grained as showing step-by-step traces of events (like HTTP access logs, pod state changes, controller actions, or scheduler decisions).

Klog

klog is the Kubernetes logging library. klog generates log messages for the Kubernetes system components.

For more information about klog configuration, see the Command line tool reference.

An example of the klog native format:

  1. I1025 00:15:15.525108       1 httplog.go:79] GET /api/v1/namespaces/kube-system/pods/metrics-server-v0.3.1-57c75779f-9p8wg: (1.512ms) 200 [pod_nanny/v0.0.0 (linux/amd64) kubernetes/$Format 10.56.1.19:51756]

Structured Logging

FEATURE STATE: Kubernetes v1.19 [alpha]

Warning:

Migration to structured log messages is an ongoing process. Not all log messages are structured in this version. When parsing log files, you must also handle unstructured log messages.

Log formatting and value serialization are subject to change.

Structured logging is a effort to introduce a uniform structure in log messages allowing for easy extraction of information, making logs easier and cheaper to store and process. New message format is backward compatible and enabled by default.

Format of structured logs:

  1. <klog header> "<message>" <key1>="<value1>" <key2>="<value2>" ...

Example:

  1. I1025 00:15:15.525108       1 controller_utils.go:116] "Pod status updated" pod="kube-system/kubedns" status="ready"

JSON log format

FEATURE STATE: Kubernetes v1.19 [alpha]

Warning:

JSON output does not support many standard klog flags. For list of unsupported klog flags, see the Command line tool reference.

Not all logs are guaranteed to be written in JSON format (for example, during process start). If you intend to parse logs, make sure you can handle log lines that are not JSON as well.

Field names and JSON serialization are subject to change.

The --logging-format=json flag changes the format of logs from klog native format to JSON format. Example of JSON log format (pretty printed):

  1. {
  2.    "ts": 1580306777.04728,
  3.    "v": 4,
  4.    "msg": "Pod status updated",
  5.    "pod":{
  6.       "name": "nginx-1",
  7.       "namespace": "default"
  8.    },
  9.    "status": "ready"
  10. }

Keys with special meaning:

  • ts - timestamp as Unix time (required, float)
  • v - verbosity (required, int, default 0)
  • err - error string (optional, string)
  • msg - message (required, string)

List of components currently supporting JSON format:

Log sanitization

FEATURE STATE: Kubernetes v1.20 [alpha]

Warning: Log sanitization might incur significant computation overhead and therefore should not be enabled in production.

The --experimental-logging-sanitization flag enables the klog sanitization filter. If enabled all log arguments are inspected for fields tagged as sensitive data (e.g. passwords, keys, tokens) and logging of these fields will be prevented.

List of components currently supporting log sanitization:

  • kube-controller-manager
  • kube-apiserver
  • kube-scheduler
  • kubelet

Note: The Log sanitization filter does not prevent user workload logs from leaking sensitive data.

Log verbosity level

The -v flag controls log verbosity. Increasing the value increases the number of logged events. Decreasing the value decreases the number of logged events. Increasing verbosity settings logs increasingly less severe events. A verbosity setting of 0 logs only critical events.

Log location

There are two types of system components: those that run in a container and those that do not run in a container. For example:

  • The Kubernetes scheduler and kube-proxy run in a container.
  • The kubelet and container runtime, for example Docker, do not run in containers.

On machines with systemd, the kubelet and container runtime write to journald. Otherwise, they write to .log files in the /var/log directory. System components inside containers always write to .log files in the /var/log directory, bypassing the default logging mechanism. Similar to the container logs, you should rotate system component logs in the /var/log directory. In Kubernetes clusters created by the kube-up.sh script, log rotation is configured by the logrotate tool. The logrotate tool rotates logs daily, or once the log size is greater than 100MB.

What’s next