Using Podman instead of Docker

Podman has an Docker API compatibility layer. k3d uses the Docker API and is compatible with Podman v4 and higher.

Podman support is experimental

k3d is not guaranteed to work with Podman. If you find a bug, do help by filing an issue

Tested with podman version:

  1. Client:       Podman Engine
  2. Version:      4.3.1
  3. API Version:  4.3.1

Using Podman

Ensure the Podman system socket is available:

  1. sudo systemctl enable --now podman.socket
  2. # or to start the socket daemonless
  3. # sudo podman system service --time=0 &

Disable timeout for podman service:
See the podman-system-service (1) man page for more information.

  1. mkdir -p /etc/containers/containers.conf.d
  2. echo 'service_timeout=0' > /etc/containers/containers.conf.d/timeout.conf

To point k3d at the right Docker socket, create a symbolic link:

  1. sudo ln -s /run/podman/podman.sock /var/run/docker.sock
  2. # or install your system podman-docker if available
  3. sudo k3d cluster create

Alternatively, set DOCKER_HOST when running k3d:

  1. export DOCKER_HOST=unix:///run/podman/podman.sock
  2. export DOCKER_SOCK=/run/podman/podman.sock
  3. sudo --preserve-env=DOCKER_HOST --preserve-env=DOCKER_SOCK k3d cluster create

Using rootless Podman

Ensure the Podman user socket is available:

  1. systemctl --user enable --now podman.socket
  2. # or podman system service --time=0 &

Set DOCKER_HOST when running k3d:

  1. XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR:-/run/user/$(id -u)}
  2. export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock
  3. export DOCKER_SOCK=$XDG_RUNTIME_DIR/podman/podman.sock
  4. k3d cluster create

Using cgroup (v2)

By default, a non-root user can only get memory controller and pids controller to be delegated.

To run properly we need to enable CPU, CPUSET, and I/O delegation

Make sure you’re running cgroup v2

If /sys/fs/cgroup/cgroup.controllers is present on your system, you are using v2, otherwise you are using v1.

  1. mkdir -p /etc/systemd/system/user@.service.d
  2. cat > /etc/systemd/system/user@.service.d/delegate.conf <<EOF
  3. [Service]
  4. Delegate=cpu cpuset io memory pids
  5. EOF
  6. systemctl daemon-reload

Reference: https://rootlesscontaine.rs/getting-started/common/cgroup2/#enabling-cpu-cpuset-and-io-delegation

Using remote Podman

Start Podman on the remote host, and then set DOCKER_HOST when running k3d:

  1. export DOCKER_HOST=ssh://username@hostname
  2. export DOCKER_SOCK=/run/user/1000/podman/podman.sock
  3. k3d cluster create

macOS

Initialize a podman machine if not done already

  1. podman machine init

Or start an already existing podman machine

  1. podman machine start

Grab connection details

  1. podman system connection ls
  2. Name                         URI                                                         Identity                                      Default
  3. podman-machine-default       ssh://core@localhost:53685/run/user/501/podman/podman.sock  /Users/myusername/.ssh/podman-machine-default  true
  4. podman-machine-default-root  ssh://root@localhost:53685/run/podman/podman.sock           /Users/myusername/.ssh/podman-machine-default  false

Edit your OpenSSH config file to specify the IdentityFile

  1. vim ~/.ssh/config
  3. Host localhost
  4.     IdentityFile /Users/myusername/.ssh/podman-machine-default

Rootless mode

Delegate the cpuset cgroup controller to the user’s systemd slice, export the docker environment variables referenced above for the non-root connection, and create the cluster:

  1. podman machine ssh bash -e <<EOF
  2.   printf '[Service]\nDelegate=cpuset\n' | sudo tee /etc/systemd/system/user@.service.d/k3d.conf
  3.   sudo systemctl daemon-reload
  4.   sudo systemctl restart "user@\${UID}"
  5. EOF
  7. export DOCKER_HOST=ssh://core@localhost:53685
  8. export DOCKER_SOCKET=/run/user/501/podman/podman.sock
  9. k3d cluster create --k3s-arg '--kubelet-arg=feature-gates=KubeletInUserNamespace=true@server:*'

Rootful mode

Export the docker environment variables referenced above for the root connection and create the cluster:

  1. export DOCKER_HOST=ssh://root@localhost:53685
  2. export DOCKER_SOCK=/run/podman/podman.sock
  3. k3d cluster create

Podman network

The default podman network has dns disabled. To allow k3d cluster nodes to communicate with dns a new network must be created.

  1. podman network create k3d
  2. podman network inspect k3d -f '{{ .DNSEnabled }}'
  3. true

Creating local registries

Because Podman does not have a default “bridge” network, you have to specify a network using the --default-network flag when creating a local registry:

  1. k3d registry create --default-network podman mycluster-registry

To use this registry with a cluster, pass the --registry-use flag:

  1. k3d cluster create --registry-use mycluster-registry mycluster

Incompatibility with --registry-create

Because --registry-create assumes the default network to be “bridge”, avoid --registry-create when using Podman. Instead, always create a registry before creating a cluster.

Missing cpuset cgroup controller

If you experince an error regarding missing cpuset cgroup controller, ensure the user unit xdg-document-portal.service is disabled by running systemctl --user stop xdg-document-portal.service. See this issue

Last update: August 3, 2023