Node Problem Detector

Overview

The Node Problem Detector monitors the health of your nodes by finding certain problems and reporting these problems to the API server. The detector runs as a daemonset on each node.

The Node Problem Detector is a Technology Preview feature only.

The Node Problem Detector reads system logs and watches for specific entries and makes these problems visible to the control plane, which you can view using OKD commands, such as oc get node and oc get event. You could then take action to correct these problems as appropriate or capture the messages using a tool of your choice, such as the OKD log monitoring. Detected problems can be in one of the following categories:

  • NodeCondition: A permanent problem that makes the node unavailable for pods. The node condition will not be cleared until the host is rebooted.

  • Event: A temporary problem that has limited impact on a node, but is informative.

The Node Problem Detector can detect:

  • container runtime issues:

    • unresponsive runtime daemons

  • hardware issues:

    • bad CPU

    • bad memory

    • bad disk

  • kernel issues:

    • kernel deadlock conditions

    • corrupted file systems

    • unresponsive runtime daemons

  • infrastructure daemon issues:

    • NTP service outages

Example Node Problem Detector Output

The following examples show output from the Node Problem Detector watching for kernel deadlock node condition on a specific node. The command uses oc get node to watch a specific node filtering for a KernelDeadlock entry in a log.

  1. # oc get node <node> -o yaml | grep -B5 KernelDeadlock

Sample Node Problem Detector output with no issues

  1. message: kernel has no deadlock
  2. reason: KernelHasNoDeadlock
  3. status: false
  4. type: KernelDeadLock

Sample output for KernelDeadLock condition

  1. message: task docker:1234 blocked for more than 120 seconds
  2. reason: DockerHung
  3. status: true
  4. type: KernelDeadLock

This example shows output from the Node Problem Detector watching for events on a node. The following command uses oc get event against the default project watching for events listed in the kernel-monitor.json section of the Node Problem Detector configuration map.

  1. # oc get event -n default --field-selector=source=kernel-monitor --watch

Sample output showing events on nodes

  1. LAST SEEN                       FIRST SEEN                    COUNT NAME     KIND  SUBOBJECT TYPE    REASON      SOURCE                   MESSAGE
  2. 2018-06-27 09:08:27 -0400 EDT   2018-06-27 09:08:27 -0400 EDT 1     my-node1 node            Warning TaskHunk    kernel-monitor.my-node1  docker:1234 blocked for more than 300 seconds
  3. 2018-06-27 09:08:27 -0400 EDT   2018-06-27 09:08:27 -0400 EDT 3     my-node2 node            Warning KernelOops  kernel-monitor.my-node2  BUG: unable to handle kernel NULL pointer deference at nowhere
  4. 2018-06-27 09:08:27 -0400 EDT   2018-06-27 09:08:27 -0400 EDT 1     my-node1 node            Warning KernelOops  kernel-monitor.my-node2  divide error 0000 [#0] SMP

The Node Problem Detector consumes resources. If you use the Node Problem Detector, make sure you have enough nodes to balance cluster performance.

Installing the Node Problem Detector

If openshift_node_problem_detector_install was set to true in the /etc/ansible/hosts inventory file, the installation creates a Node Problem Detector daemonset by default and creates a project for the detector, called openshift-node-problem-detector.

Because the Node Problem Detector is in Technology Preview, the openshift_node_problem_detector_install is set to false by default. You must manually change the parameter to true when installing the Node Problem Detector.

If the Node Problem Detector is not installed, change to the playbook directory and run the openshift-node-problem-detector/config.yml playbook to install Node Problem Detector:

  1. $ cd /usr/share/ansible/openshift-ansible
  2. $ ansible-playbook playbooks/openshift-node-problem-detector/config.yml

Customizing Detected Conditions

You can configure the Node Problem Detector to watch for any log string by editing the Node Problem Detector configuration map.

Sample Node Problem Detector Configuration Map

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: node-problem-detector
  5. data:
  6.   docker-monitor.json: |  (1)
  7.     {
  8.         "plugin": "journald", (2)
  9.         "pluginConfig": {
  10.                 "source": "docker"
  11.         },
  12.         "logPath": "/host/log/journal", (3)
  13.         "lookback": "5m",
  14.         "bufferSize": 10,
  15.         "source": "docker-monitor",
  16.         "conditions": [],
  17.         "rules": [              (4)
  18.                 {
  19.                         "type": "temporary", (5)
  20.                         "reason": "CorruptDockerImage", (6)
  21.                         "pattern": "Error trying v2 registry: failed to register layer: rename /var/lib/docker/image/(.+) /var/lib/docker/image/(.+): directory not empty.*" (7)
  22.                 }
  23.         ]
  24.     }
  25.   kernel-monitor.json: |  (8)
  26.     {
  27.         "plugin": "journald", (2)
  28.         "pluginConfig": {
  29.                 "source": "kernel"
  30.         },
  31.         "logPath": "/host/log/journal", (3)
  32.         "lookback": "5m",
  33.         "bufferSize": 10,
  34.         "source": "kernel-monitor",
  35.         "conditions": [                 (4)
  36.                 {
  37.                         "type": "KernelDeadlock", (5)
  38.                         "reason": "KernelHasNoDeadlock", (6)
  39.                         "message": "kernel has no deadlock"  (7)
  40.                 }
  41.         ],
  42.         "rules": [
  43.                 {
  44.                         "type": "temporary",
  45.                         "reason": "OOMKilling",
  46.                         "pattern": "Kill process \\d+ (.+) score \\d+ or sacrifice child\\nKilled process \\d+ (.+) total-vm:\\d+kB, anon-rss:\\d+kB, file-rss:\\d+kB"
  47.                 },
  48.                 {
  49.                         "type": "temporary",
  50.                         "reason": "TaskHung",
  51.                         "pattern": "task \\S+:\\w+ blocked for more than \\w+ seconds\\."
  52.                 },
  53.                 {
  54.                         "type": "temporary",
  55.                         "reason": "UnregisterNetDevice",
  56.                         "pattern": "unregister_netdevice: waiting for \\w+ to become free. Usage count = \\d+"
  57.                 },
  58.                 {
  59.                         "type": "temporary",
  60.                         "reason": "KernelOops",
  61.                         "pattern": "BUG: unable to handle kernel NULL pointer dereference at .*"
  62.                 },
  63.                 {
  64.                         "type": "temporary",
  65.                         "reason": "KernelOops",
  66.                         "pattern": "divide error: 0000 \\[#\\d+\\] SMP"
  67.                 },
  68.                 {
  69.                         "type": "permanent",
  70.                         "condition": "KernelDeadlock",
  71.                         "reason": "AUFSUmountHung",
  72.                         "pattern": "task umount\\.aufs:\\w+ blocked for more than \\w+ seconds\\."
  73.                 },
  74.                 {
  75.                         "type": "permanent",
  76.                         "condition": "KernelDeadlock",
  77.                         "reason": "DockerHung",
  78.                         "pattern": "task docker:\\w+ blocked for more than \\w+ seconds\\."
  79.                 }
  80.         ]
  81.     }
1Rules and conditions that apply to container images.
2Monitoring services, in a comma-separated list.
3Path to the monitoring service log.
4List of events to be monitored.
5Label to indicate the error is an event (temporary) or NodeCondition (permanent).
6Text message to describe the error.
7Error message that the Node Problem Detector watches for.
8Rules and conditions that apply to the kernel.

To configure the Node Problem Detector, add or remove problem conditions and events.

  1. Edit the Node Problem Detector configuration map with a text editor.

    1. oc edit configmap -n openshift-node-problem-detector node-problem-detector

  2. Remove, add, or edit any node conditions or events as needed.

    1. {
    2.        "type": <`temporary` or `permanent`>,
    3.        "reason": <free-form text describing the error>,
    4.        "pattern": <log message to watch for>
    5. },

    For example:

    1. {
    2.        "type": "temporary",
    3.        "reason": "UnregisterNetDevice",
    4.        "pattern": "unregister_netdevice: waiting for \\w+ to become free. Usage count = \\d+"
    5. },

  3. Restart running pods to apply the changes. To restart pods, you can delete all existing pods:

    1. # oc delete pods -n openshift-node-problem-detector -l name=node-problem-detector

  4. To display Node Problem Detector output to standard output (stdout) and standard error (stderr) add the following to the configuration map:

    1. spec:
    2.   template:
    3.     spec:
    4.       containers:
    5.       - name: node-problem-detector
    6.         command:
    7.         - node-problem-detector
    8.         - --alsologtostderr=true (1)
    9.         - --log_dir="/tmp" (2)
    10.         - --system-log-monitors=/etc/npd/kernel-monitor.json,/etc/npd/docker-monitor.json (3)
    1Sends the output to standard output (stdout).
    2Path to the error log.
    3Comma-separated path to the plug-in configuration files.

Verifying that the Node Problem Detector is Running

To verify that the Node Problem Detector is active:

  • Run the following command to get the name of the Problem Node Detector pod:

    1. # oc get pods -n openshift-node-problem-detector
    3. NAME                          READY     STATUS    RESTARTS   AGE
    4. node-problem-detector-8z8r8   1/1       Running   0          1h
    5. node-problem-detector-nggjv   1/1       Running   0          1h

  • Run the following command to view log information on the Problem Node Detector pod:

    1. # oc logs -n openshift-node-problem-detector <pod_name>

    The output should be similar to the following:

    1. # oc logs -n openshift-node-problem-detector node-problem-detector-c6kng
    2. I0416 23:22:00.641354       1 log_monitor.go:63] Finish parsing log monitor config file: {WatcherConfig:{Plugin:journald PluginConfig:map[source:kernel] LogPath:/host/log/journal Lookback:5m} BufferSize:10 Source:kernel-monitor DefaultConditions:[{Type:KernelDeadlock Status:false Transition:0001-01-01 00:00:00 +0000 UTC Reason:KernelHasNoDeadlock Message:kernel has no deadlock}]

  • Test the Node Problem Detector by simulating an event on the node:

    1. # echo "kernel: divide error: 0000 [#0] SMP." >> /dev/kmsg

  • Test the Node Problem Detector by simulating a condition on the node:

    1. # echo "kernel: task docker:7 blocked for more than 300 seconds." >> /dev/kmsg

Uninstall the Node Problem Detector

To uninstall the Node Problem Detector:

  1. Add following options in Ansible inventory file:

    1. [OSEv3:vars]
    2. openshift_node_problem_detector_state=absent

  2. Change to the playbook directory and run the config.yml Ansible playbook:

    1. $ cd /usr/share/ansible/openshift-ansible
    2. $ ansible-playbook playbooks/openshift-node-problem-detector/config.yml