Configure Access to Multiple Clusters

This page shows how to configure access to multiple clusters by using configuration files. After your clusters, users, and contexts are defined in one or more configuration files, you can quickly switch between clusters by using the kubectl config use-context command.

Note: A file that is used to configure access to a cluster is sometimes called a kubeconfig file. This is a generic way of referring to configuration files. It does not mean that there is a file named kubeconfig.

Warning: Only use kubeconfig files from trusted sources. Using a specially-crafted kubeconfig file could result in malicious code execution or file exposure. If you must use an untrusted kubeconfig file, inspect it carefully first, much as you would a shell script.

Before you begin

You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

To check that kubectl is installed, run kubectl version --client. The kubectl version should be within one minor version of your cluster’s API server.

Define clusters, users, and contexts

Suppose you have two clusters, one for development work and one for scratch work. In the development cluster, your frontend developers work in a namespace called frontend, and your storage developers work in a namespace called storage. In your scratch cluster, developers work in the default namespace, or they create auxiliary namespaces as they see fit. Access to the development cluster requires authentication by certificate. Access to the scratch cluster requires authentication by username and password.

Create a directory named config-exercise. In your config-exercise directory, create a file named config-demo with this content:

  1. apiVersion: v1
  2. kind: Config
  3. preferences: {}
  5. clusters:
  6. - cluster:
  7.   name: development
  8. - cluster:
  9.   name: scratch
  11. users:
  12. - name: developer
  13. - name: experimenter
  15. contexts:
  16. - context:
  17.   name: dev-frontend
  18. - context:
  19.   name: dev-storage
  20. - context:
  21.   name: exp-scratch

A configuration file describes clusters, users, and contexts. Your config-demo file has the framework to describe two clusters, two users, and three contexts.

Go to your config-exercise directory. Enter these commands to add cluster details to your configuration file:

  1. kubectl config --kubeconfig=config-demo set-cluster development --server=https://1.2.3.4 --certificate-authority=fake-ca-file
  2. kubectl config --kubeconfig=config-demo set-cluster scratch --server=https://5.6.7.8 --insecure-skip-tls-verify

Add user details to your configuration file:

Caution: Storing passwords in Kubernetes client config is risky. A better alternative would be to use a credential plugin and store them separately. See: client-go credential plugins

  1. kubectl config --kubeconfig=config-demo set-credentials developer --client-certificate=fake-cert-file --client-key=fake-key-seefile
  2. kubectl config --kubeconfig=config-demo set-credentials experimenter --username=exp --password=some-password

Note:

  • To delete a user you can run kubectl --kubeconfig=config-demo config unset users.<name>
  • To remove a cluster, you can run kubectl --kubeconfig=config-demo config unset clusters.<name>
  • To remove a context, you can run kubectl --kubeconfig=config-demo config unset contexts.<name>

Add context details to your configuration file:

  1. kubectl config --kubeconfig=config-demo set-context dev-frontend --cluster=development --namespace=frontend --user=developer
  2. kubectl config --kubeconfig=config-demo set-context dev-storage --cluster=development --namespace=storage --user=developer
  3. kubectl config --kubeconfig=config-demo set-context exp-scratch --cluster=scratch --namespace=default --user=experimenter

Open your config-demo file to see the added details. As an alternative to opening the config-demo file, you can use the config view command.

  1. kubectl config --kubeconfig=config-demo view

The output shows the two clusters, two users, and three contexts:

  1. apiVersion: v1
  2. clusters:
  3. - cluster:
  4.     certificate-authority: fake-ca-file
  5.     server: https://1.2.3.4
  6.   name: development
  7. - cluster:
  8.     insecure-skip-tls-verify: true
  9.     server: https://5.6.7.8
  10.   name: scratch
  11. contexts:
  12. - context:
  13.     cluster: development
  14.     namespace: frontend
  15.     user: developer
  16.   name: dev-frontend
  17. - context:
  18.     cluster: development
  19.     namespace: storage
  20.     user: developer
  21.   name: dev-storage
  22. - context:
  23.     cluster: scratch
  24.     namespace: default
  25.     user: experimenter
  26.   name: exp-scratch
  27. current-context: ""
  28. kind: Config
  29. preferences: {}
  30. users:
  31. - name: developer
  32.   user:
  33.     client-certificate: fake-cert-file
  34.     client-key: fake-key-file
  35. - name: experimenter
  36.   user:
  37.     # Documentation note (this comment is NOT part of the command output).
  38.     # Storing passwords in Kubernetes client config is risky.
  39.     # A better alternative would be to use a credential plugin
  40.     # and store the credentials separately.
  41.     # See https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins
  42.     password: some-password
  43.     username: exp

The fake-ca-file, fake-cert-file and fake-key-file above are the placeholders for the pathnames of the certificate files. You need to change these to the actual pathnames of certificate files in your environment.

Sometimes you may want to use Base64-encoded data embedded here instead of separate certificate files; in that case you need to add the suffix -data to the keys, for example, certificate-authority-data, client-certificate-data, client-key-data.

Each context is a triple (cluster, user, namespace). For example, the dev-frontend context says, “Use the credentials of the developer user to access the frontend namespace of the development cluster”.

Set the current context:

  1. kubectl config --kubeconfig=config-demo use-context dev-frontend

Now whenever you enter a kubectl command, the action will apply to the cluster, and namespace listed in the dev-frontend context. And the command will use the credentials of the user listed in the dev-frontend context.

To see only the configuration information associated with the current context, use the --minify flag.

  1. kubectl config --kubeconfig=config-demo view --minify

The output shows configuration information associated with the dev-frontend context:

  1. apiVersion: v1
  2. clusters:
  3. - cluster:
  4.     certificate-authority: fake-ca-file
  5.     server: https://1.2.3.4
  6.   name: development
  7. contexts:
  8. - context:
  9.     cluster: development
  10.     namespace: frontend
  11.     user: developer
  12.   name: dev-frontend
  13. current-context: dev-frontend
  14. kind: Config
  15. preferences: {}
  16. users:
  17. - name: developer
  18.   user:
  19.     client-certificate: fake-cert-file
  20.     client-key: fake-key-file

Now suppose you want to work for a while in the scratch cluster.

Change the current context to exp-scratch:

  1. kubectl config --kubeconfig=config-demo use-context exp-scratch

Now any kubectl command you give will apply to the default namespace of the scratch cluster. And the command will use the credentials of the user listed in the exp-scratch context.

View configuration associated with the new current context, exp-scratch.

  1. kubectl config --kubeconfig=config-demo view --minify

Finally, suppose you want to work for a while in the storage namespace of the development cluster.

Change the current context to dev-storage:

  1. kubectl config --kubeconfig=config-demo use-context dev-storage

View configuration associated with the new current context, dev-storage.

  1. kubectl config --kubeconfig=config-demo view --minify

Create a second configuration file

In your config-exercise directory, create a file named config-demo-2 with this content:

  1. apiVersion: v1
  2. kind: Config
  3. preferences: {}
  5. contexts:
  6. - context:
  7.     cluster: development
  8.     namespace: ramp
  9.     user: developer
  10.   name: dev-ramp-up

The preceding configuration file defines a new context named dev-ramp-up.

Set the KUBECONFIG environment variable

See whether you have an environment variable named KUBECONFIG. If so, save the current value of your KUBECONFIG environment variable, so you can restore it later. For example:

Linux

  1. export KUBECONFIG_SAVED="$KUBECONFIG"

Windows PowerShell

  1. $Env:KUBECONFIG_SAVED=$ENV:KUBECONFIG

The KUBECONFIG environment variable is a list of paths to configuration files. The list is colon-delimited for Linux and Mac, and semicolon-delimited for Windows. If you have a KUBECONFIG environment variable, familiarize yourself with the configuration files in the list.

Temporarily append two paths to your KUBECONFIG environment variable. For example:

Linux

  1. export KUBECONFIG="${KUBECONFIG}:config-demo:config-demo-2"

Windows PowerShell

  1. $Env:KUBECONFIG=("config-demo;config-demo-2")

In your config-exercise directory, enter this command:

  1. kubectl config view

The output shows merged information from all the files listed in your KUBECONFIG environment variable. In particular, notice that the merged information has the dev-ramp-up context from the config-demo-2 file and the three contexts from the config-demo file:

  1. contexts:
  2. - context:
  3.     cluster: development
  4.     namespace: frontend
  5.     user: developer
  6.   name: dev-frontend
  7. - context:
  8.     cluster: development
  9.     namespace: ramp
  10.     user: developer
  11.   name: dev-ramp-up
  12. - context:
  13.     cluster: development
  14.     namespace: storage
  15.     user: developer
  16.   name: dev-storage
  17. - context:
  18.     cluster: scratch
  19.     namespace: default
  20.     user: experimenter
  21.   name: exp-scratch

For more information about how kubeconfig files are merged, see Organizing Cluster Access Using kubeconfig Files

Explore the $HOME/.kube directory

If you already have a cluster, and you can use kubectl to interact with the cluster, then you probably have a file named config in the $HOME/.kube directory.

Go to $HOME/.kube, and see what files are there. Typically, there is a file named config. There might also be other configuration files in this directory. Briefly familiarize yourself with the contents of these files.

Append $HOME/.kube/config to your KUBECONFIG environment variable

If you have a $HOME/.kube/config file, and it’s not already listed in your KUBECONFIG environment variable, append it to your KUBECONFIG environment variable now. For example:

Linux

  1. export KUBECONFIG="${KUBECONFIG}:${HOME}/.kube/config"

Windows Powershell

  1. $Env:KUBECONFIG="$Env:KUBECONFIG;$HOME\.kube\config"

View configuration information merged from all the files that are now listed in your KUBECONFIG environment variable. In your config-exercise directory, enter:

  1. kubectl config view

Clean up

Return your KUBECONFIG environment variable to its original value. For example:

Linux

  1. export KUBECONFIG="$KUBECONFIG_SAVED"

Windows PowerShell

  1. $Env:KUBECONFIG=$ENV:KUBECONFIG_SAVED

Check the subject represented by the kubeconfig

It is not always obvious what attributes (username, groups) you will get after authenticating to the cluster. It can be even more challenging if you are managing more than one cluster at the same time.

There is a kubectl alpha subcommand command to check subject attributes, such as username, for your selected Kubernetes client context: kubectl alpha auth whoami.

Read API access to authentication information for a client to learn about this in more detail.

What’s next