Troubleshooting

It is important to recognise that things can go wrong. But MicroK8s gives you tools to help work out what has gone wrong, as detailed below. Be sure to check out the common issues section for help resolving the most frequently encountered problems.

Checking logs

If a pod is not behaving as expected, the first port of call should be the logs.

First determine the resource identifier for the pod:

  1. microk8s kubectl get pods

This will list the currently available pods, for example:

  1. NAME                                 READY   STATUS             RESTARTS   AGE
  2. mk8s-redis-7647889b6d-vjwqm          1/1     Running            0          2m24s

You can then use kubectl to view the log. For example, for the simple redis pod above:

  1. microk8s kubectl logs mk8s-redis-7647889b6d-vjwqm

Examining the configuration

If the problem you are experiencing indicates a problem with the configuration of the Kubernetes components themselves, it could be helpful to examine the arguments used to run these components.

These are available in the directory ${SNAP_DATA}/args, which on Ubuntu should point to /var/snap/microk8s/current.
Note that the $SNAP_DATA environment variable itself is only available to the running snap. For more information on the snap environment, check the snap documentation.

Using the built-in inspection tool

MicroK8s ships with a script to compile a complete report on MicroK8s and the system which it is running on. This is essential for bug reports, but is also a useful way of confirming the system is (or isn’t) working and collecting all the relevant data in one place.

To run the inspection tool, enter the command (admin privilege is required to collect all the data):

  1. sudo microk8s inspect

You should see output similar to the following:

  1. Inspecting services
  2.   Service snap.microk8s.daemon-cluster-agent is running
  3.   Service snap.microk8s.daemon-flanneld is running
  4.   Service snap.microk8s.daemon-containerd is running
  5.   Service snap.microk8s.daemon-apiserver is running
  6.   Service snap.microk8s.daemon-apiserver-kicker is running
  7.   Service snap.microk8s.daemon-proxy is running
  8.   Service snap.microk8s.daemon-kubelet is running
  9.   Service snap.microk8s.daemon-scheduler is running
  10.   Service snap.microk8s.daemon-controller-manager is running
  11.   Service snap.microk8s.daemon-etcd is running
  12.   Copy service arguments to the final report tarball
  13. Inspecting AppArmor configuration
  14. Gathering system information
  15.   Copy processes list to the final report tarball
  16.   Copy snap list to the final report tarball
  17.   Copy VM name (or none) to the final report tarball
  18.   Copy disk usage information to the final report tarball
  19.   Copy memory usage information to the final report tarball
  20.   Copy server uptime to the final report tarball
  21.   Copy current linux distribution to the final report tarball
  22.   Copy openSSL information to the final report tarball
  23.   Copy network configuration to the final report tarball
  24. Inspecting kubernetes cluster
  25.   Inspect kubernetes cluster
  27. Building the report tarball
  28.   Report tarball is at /var/snap/microk8s/1031/inspection-report-20191104_153950.tar.gz

This confirms the services that are running, and the resultant report file can be viewed to get a detailed look at every aspect of the system.

Common issues

Node is not ready when RBAC is enabled…

Ensure the hostname of your machine name does not contain capital letters or underscores. Kubernetes normalizes the machine name causing its registration to fail.

To fix this you can change the hostname or use the --hostname-override argument in kubelet’s configuration in /var/snap/microk8s/current/args/kubelet.

My dns and dashboard pods are CrashLooping…

The cni network plugin used by MicroK8s creates a vxlan.calico interface (cbr0 on pre v1.16 releases or cni0 in pre v1.19 releases and non-HA deployments) when the first pod is created.

If you have ufw enabled, you’ll need to allow traffic on this interface:

  1. sudo ufw allow in on vxlan.calico && sudo ufw allow out on vxlan.calico

My pods can’t reach the internet or each other (but my MicroK8s host machine can)…

Make sure packets to/from the pod network interface can be forwarded to/from the default interface on the host via the iptables tool. Such changes can be made persistent by installing the iptables-persistent package:

  1.    sudo iptables -P FORWARD ACCEPT
  2.    sudo apt-get install iptables-persistent

or, if using ufw:

  1.    sudo ufw default allow routed

The MicroK8s inspect command can be used to check the firewall configuration:

  1.    microk8s inspect

A warning will be shown if the firewall is not forwarding traffic.

My log collector is not collecting any logs…

By default container logs are located in /var/log/pods/{id}. You have to mount this location in your log collector for that to work. Following is an example diff for fluent-bit:

  1. @@ -36,6 +36,9 @@
  2.             - name: varlibdockercontainers
  3.               mountPath: /var/lib/docker/containers
  4.               readOnly: true
  5.    +        - name: varlibdockercontainers
  6.    +          mountPath: /var/snap/microk8s/common/var/lib/containerd/
  7.    +          readOnly: true
  8.             - name: fluent-bit-config
  9.               mountPath: /fluent-bit/etc/
  10.           terminationGracePeriodSeconds: 10
  11.    @@ -45,7 +48,7 @@
  12.               path: /var/log
  13.           - name: varlibdockercontainers
  14.             hostPath:
  15.    -          path: /var/lib/docker/containers
  16.    +          mountPath: /var/snap/microk8s/common/var/lib/containerd/
  17.           - name: fluent-bit-config
  18.             configMap:
  19.               name: fluent-bit-config
  21. ``` **My pods are not starting and I use ZFS...**
  23. Microk8s switched to `containerd` as its container runtime in release 492. When run on ZFS, `containerd` must be configured to use ZFS snapshots. Presently neither Microk8s nor `containerd` perform this automatically so you must manually update the configuration. Instructions on how to do this are documented [here](https://github.com/ubuntu/microk8s/issues/401#issuecomment-480945986).
  25. **My home directory is not in `/home` or is on NFS and I can't get Microk8s to work...**
  27. While not strictly a Microk8s issue, snaps generally do not work out of the box if your home directory is mounted via NFS, or if it is not located directly under `/home`. See `snapd` bugs [#1662552](https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1662552) and [#1620771](https://bugs.launchpad.net/snappy/+bug/1620771) for further information and possible workarounds.
  29. **I need to recover my HA cluster**
  31. In normal use, a MicroK8s HA cluster is self healing. There may be occasions when testing edge versions or mixing releases that the cluster may need to be recovered. This [docs page](https://discuss.kubernetes.io/t/recovery-of-ha-microk8s-clusters/12931) details the procedure.
  33. **Raspberry Pi and systems with low disk performance**
  35. The symptoms you may observe vary. You may experience the API server being slow, crashing or forming an unstable multi node cluster. Such problems are often traced to low performing or miss-configured disks. In the logs of the API server you will notice the data store being slow to write on disk. With `journalctl -f -u snap.microk8s.daemon-kubelite` or (for prior to v1.21) `journalctl -f -u snap.microk8s.daemon-apiserver` you will see messages such as `microk8s.daemon-kubelite[3802920]: Trace[736557743]: ---"Object stored in database" 7755ms` .
  37. To identify if the a slow disk is affecting you, you could use `hdparm` and try to write a large file with `dd`, for example, `hdparm -Tt /dev/sda` and `dd if=/dev/zero of=/tmp/test1.img bs=1G count=1`
  39. In systems such as the Raspberry Pi the issue may caused by [devices not fully implementing the UAS specification](https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=245931).
  41. In some cases, a way to mitigate the issue is to move the [journald logs on volatile storage](https://www.freedesktop.org/software/systemd/man/journald.conf.html). This is done by editing `/etc/systemd/journald.conf` setting `Storage=volatile`.
  43. Of course, you can always consider upgrading the attached storage.
  45. The issue [API Server hanging on raspberry pi · Issue #2280 · ubuntu/microk8s · GitHub](https://github.com/ubuntu/microk8s/issues/2280) demonstrates a successful debugging of this issue.
  47. **"access denied" error on Debian 9**
  49. Snapctl coming with Debian 9 is outdated. Here is how to replace it with a fresh one:

sudo snap install core sudo mv /usr/bin/snapctl /usr/bin/snapctl.old sudo ln -s /snap/core/current/usr/bin/snapctl /usr/bin/snapctl ```

Reporting a bug

If you cannot solve your issue and believe the fault may lie in MicroK8s, please file an issue on the project repository.

To help us deal effectively with issues, it is incredibly useful to include the report obtained from microk8s inspect, as well as any additional logs, and a summary of the issue.

Last updated 2 days ago. Help improve this document in the forum.