Using Config Files

The config file feature is available as of k3d v4.0.0

Introduction

Syntax & Semantics

The options defined in the config file are not 100% the same as the CLI flags.
This concerns naming and style/usage/structure, e.g.

  • --api-port is split up into a field named kubeAPI that has 3 different “child fields” (host, hostIP and hostPort)
  • k3d options are bundled in a scope named options.k3d, where --no-rollback is defined as options.k3d.disableRollback
  • repeatable flags (like --port) are reflected as YAML lists

Usage

Using a config file is as easy as putting it in a well-known place in your file system and then referencing it via flag:

  • All options in config file: k3d cluster create --config /home/me/my-awesome-config.yaml (must be .yaml/.yml)
  • With CLI override (name): k3d cluster create somename --config /home/me/my-awesome-config.yaml
  • With CLI override (extra volume): k3d cluster create --config /home/me/my-awesome-config.yaml --volume '/some/path:/some:path@server:0'

Required Fields

As of the time of writing this documentation, the config file only requires you to define two fields:

  • apiVersion to match the version of the config file that you want to use (at this time it would be apiVersion: k3d.io/v1alpha5)
  • kind to define the kind of config file that you want to use (currently we only have the Simple config)

So this would be the minimal config file, which configures absolutely nothing:

  1. apiVersion: k3d.io/v1alpha5
  2. kind: Simple

Config Options

The configuration options for k3d are continuously evolving and so is the config file (syntax) itself.
Currently, the config file is still in an Alpha-State, meaning, that it is subject to change anytime (though we try to keep breaking changes low).

Validation via JSON-Schema

k3d uses a JSON-Schema to describe the expected format and fields of the configuration file.
This schema is also used to validate a user-given config file.
This JSON-Schema can be found in the specific config version sub-directory in the repository (e.g. here for v1alpha5) and could be used to lookup supported fields or by linters to validate the config file, e.g. in your code editor.

All Options: Example

Since the config options and the config file are changing quite a bit, it’s hard to keep track of all the supported config file settings, so here’s an example showing all of them as of the time of writing:

  1. # k3d configuration file, saved as e.g. /home/me/myk3dcluster.yaml
  2. apiVersion: k3d.io/v1alpha5 # this will change in the future as we make everything more stable
  3. kind: Simple # internally, we also have a Cluster config, which is not yet available externally
  4. metadata:
  5.   name: mycluster # name that you want to give to your cluster (will still be prefixed with `k3d-`)
  6. servers: 1 # same as `--servers 1`
  7. agents: 2 # same as `--agents 2`
  8. kubeAPI: # same as `--api-port myhost.my.domain:6445` (where the name would resolve to 127.0.0.1)
  9.   host: "myhost.my.domain" # important for the `server` setting in the kubeconfig
  10.   hostIP: "127.0.0.1" # where the Kubernetes API will be listening on
  11.   hostPort: "6445" # where the Kubernetes API listening port will be mapped to on your host system
  12. image: rancher/k3s:v1.20.4-k3s1 # same as `--image rancher/k3s:v1.20.4-k3s1`
  13. network: my-custom-net # same as `--network my-custom-net`
  14. subnet: "172.28.0.0/16" # same as `--subnet 172.28.0.0/16`
  15. token: superSecretToken # same as `--token superSecretToken`
  16. volumes: # repeatable flags are represented as YAML lists
  17.   - volume: /my/host/path:/path/in/node # same as `--volume '/my/host/path:/path/in/node@server:0;agent:*'`
  18.     nodeFilters:
  19.       - server:0
  20.       - agent:*
  21. ports:
  22.   - port: 8080:80 # same as `--port '8080:80@loadbalancer'`
  23.     nodeFilters:
  24.       - loadbalancer
  25. env:
  26.   - envVar: bar=baz # same as `--env 'bar=baz@server:0'`
  27.     nodeFilters:
  28.       - server:0
  29. registries: # define how registries should be created or used
  30.   create: # creates a default registry to be used with the cluster; same as `--registry-create registry.localhost`
  31.     name: registry.localhost
  32.     host: "0.0.0.0"
  33.     hostPort: "5000"
  34.     proxy: # omit this to have a "normal" registry, set this to create a registry proxy (pull-through cache)
  35.       remoteURL: https://registry-1.docker.io # mirror the DockerHub registry
  36.       username: "" # unauthenticated
  37.       password: "" # unauthenticated
  38.     volumes:
  39.       - /some/path:/var/lib/registry # persist registry data locally
  40.   use:
  41.     - k3d-myotherregistry:5000 # some other k3d-managed registry; same as `--registry-use 'k3d-myotherregistry:5000'`
  42.   config: | # define contents of the `registries.yaml` file (or reference a file); same as `--registry-config /path/to/config.yaml`
  43.     mirrors:
  44.       "my.company.registry":
  45.         endpoint:
  46.           - http://my.company.registry:5000
  47. hostAliases: # /etc/hosts style entries to be injected into /etc/hosts in the node containers and in the NodeHosts section in CoreDNS
  48.   - ip: 1.2.3.4
  49.     hostnames: 
  50.       - my.host.local
  51.       - that.other.local
  52.   - ip: 1.1.1.1
  53.     hostnames:
  54.       - cloud.flare.dns
  55. options:
  56.   k3d: # k3d runtime settings
  57.     wait: true # wait for cluster to be usable before returning; same as `--wait` (default: true)
  58.     timeout: "60s" # wait timeout before aborting; same as `--timeout 60s`
  59.     disableLoadbalancer: false # same as `--no-lb`
  60.     disableImageVolume: false # same as `--no-image-volume`
  61.     disableRollback: false # same as `--no-Rollback`
  62.     loadbalancer:
  63.       configOverrides:
  64.         - settings.workerConnections=2048
  65.   k3s: # options passed on to K3s itself
  66.     extraArgs: # additional arguments passed to the `k3s server|agent` command; same as `--k3s-arg`
  67.       - arg: "--tls-san=my.host.domain"
  68.         nodeFilters:
  69.           - server:*
  70.     nodeLabels:
  71.       - label: foo=bar # same as `--k3s-node-label 'foo=bar@agent:1'` -> this results in a Kubernetes node label
  72.         nodeFilters:
  73.           - agent:1
  74.   kubeconfig:
  75.     updateDefaultKubeconfig: true # add new cluster to your default Kubeconfig; same as `--kubeconfig-update-default` (default: true)
  76.     switchCurrentContext: true # also set current-context to the new cluster's context; same as `--kubeconfig-switch-context` (default: true)
  77.   runtime: # runtime (docker) specific options
  78.     gpuRequest: all # same as `--gpus all`
  79.     labels:
  80.       - label: bar=baz # same as `--runtime-label 'bar=baz@agent:1'` -> this results in a runtime (docker) container label
  81.         nodeFilters:
  82.           - agent:1
  83.     ulimits:
  84.       - name: nofile
  85.         soft: 26677
  86.         hard: 26677

Tips

Config File vs. CLI Flags

k3d uses Cobra and Viper for CLI and general config handling respectively.
This automatically introduces a “config option order of priority” (precedence order):

Config Precedence Order

Source: spf13/viper#why-viper

Internal Setting > CLI Flag > Environment Variable > Config File > (k/v store >) Defaults

This means, that you can define e.g. a “base configuration file” with settings that you share across different clusters and override only the fields that differ between those clusters in your CLI flags/arguments.
For example, you use the same config file to create three clusters which only have different names and kubeAPI (--api-port) settings.

References

Last update: April 28, 2023