Upgrading from kube-up to kops

Kops let you upgrade an existing kubernetes cluster installed using kube-up, to a cluster managed bykops.

This is a slightly risky procedure, so we recommend backing up important data before proceeding. Take a snapshot of your EBS volumes; export all your data from kubectl etc.

Limitations:

  • kops splits etcd onto two volumes now: main and events. We will keep the main data, but you will lose your events history.

Overview

There are a few steps to upgrade a kubernetes cluster:

  • First you import the existing cluster state, so you can see and edit the configuration
  • You verify the cluster configuration
  • You move existing AWS resources to your new cluster
  • You bring up the new cluster
  • You can then delete the old cluster

Importing the existing cluster

The import cluster command reverse engineers an existing cluster, and creates a clusterconfiguration.

Make sure you have set export KOPS_STATE_STORE=s3://<mybucket>

Then import the cluster; setting —name and —region to match the old cluster. If you're not sureof the old cluster name, you can find it by looking at the KubernetesCluster tag on your AWS resources.

  1. export OLD_NAME=kubernetes
  2. export REGION=us-west-2
  3. kops import cluster --region ${REGION} --name ${OLD_NAME}

Verify the cluster configuration

Now have a look at the cluster configuration, to make sure it looks right. If it doesn't, pleaseopen an issue.

  1. kops get cluster ${OLD_NAME} -oyaml
  2. ````
  3.  
  4. ## Move resources to a new cluster
  5.  
  6. The upgrade moves some resources so they will be adopted by the new cluster.  There are a number of things
  7. this step does:
  8.  
  9. * It resizes existing autoscaling groups to size 0
  10. * It will stop the existing master
  11. * It detaches the master EBS volume from the master
  12. * It re-tags resources to associate them with the new cluster: volumes, ELBs
  13. * It re-tags the VPC to associate it with the new cluster
  14.  
  15. The upgrade procedure forces you to choose a new cluster name (e.g. `k8s.mydomain.com`)

export NEW_NAME=k8s.mydomain.comkops toolbox convert-imported —newname ${NEW_NAME} —name ${OLD_NAME}

  1. If you now list the clusters, you should see both the old cluster & the new cluster

kops get clusters

  1. You can also list the instance groups: `kops get ig --name ${NEW_NAME}`
  2.  
  3. ## Import the SSH public key
  4.  
  5. The SSH public key is not easily retrieved from the old cluster, so you must add it:

kops create secret —name ${NEW_NAME} sshpublickey admin -i ~/.ssh/id_rsa.pub

  1. ## Bring up the new cluster
  2.  
  3. Use the update command to bring up the new cluster:

kops update cluster ${NEW_NAME}

  1. Things to check are that it is reusing the existing volume for the _main_ etcd cluster (but not the events clusters).
  2.  
  3. And then when you are happy:

kops update cluster ${NEW_NAME} —yes

  1. ## Export kubecfg settings to access the new cluster
  2.  
  3. You can export a kubecfg (although update cluster did this automatically): `kops export kubecfg ${NEW_NAME}`
  4.  
  5.  
  6. ## Workaround for secret import failure
  7.  
  8. The import procedure tries to preserve the CA certificates, but unfortunately this isn't supported
  9. in kubernetes until [#34029](https://github.com/kubernetes/kubernetes/pull/34029) ships (should be
  10. in 1.5).
  11.  
  12. So you will need to delete the service-accounts, so they can be recreated with the correct keys.
  13.  
  14. Unfortunately, until you do this, some services (most notably internal & external DNS) will not work.
  15. Because of that you must SSH to the master to do this repair.
  16.  
  17. You can get the public IP address of the master from the AWS console, or by doing this:

aws ec2 —region $REGION describe-instances \ —filter Name=tag:KubernetesCluster,Values=${NEW_NAME} \ Name=tag-key,Values=k8s.io/role/master \ Name=instance-state-name,Values=running \ —query Reservations[].Instances[].PublicIpAddress \ —output text

  1. Then `ssh admin@<ip>` (the SSH key will be the one you added above, i.e. `~/.ssh/id_rsa.pub`), and run:
  2.  
  3. First check that the apiserver is running:

kubectl get nodes

  1. You should see only one node (the master).  Then run

NS=kubectl get namespaces -o 'jsonpath={.items[*].metadata.name}'for i in ${NS}; do kubectl get secrets —namespace=${i} —no-headers | grep "kubernetes.io/service-account-token" | awk '{print $1}' | xargs -I {} kubectl delete secret —namespace=$i {}; donesleep 60 # Allow for new secrets to be createdkubectl delete pods -lk8s-app=dns-controller —namespace=kube-systemkubectl delete pods -lk8s-app=kube-dns —namespace=kube-system

  1. You probably also want to delete the imported DNS services from prior versions:

kubectl delete rc -lk8s-app=kube-dns —namespace=kube-system # Will work for k8s <= 1.4kubectl delete deployment —namespace=kube-system kube-dns # Will work for k8s >= 1.5

  1. Within a few minutes the new cluster should be running.
  2.  
  3. Try `kubectl get nodes --show-labels`, `kubectl get pods --all-namespaces` etc until you are sure that all is well.
  4.  
  5. This should work even without being SSH-ed into the master, although it can take a few minutes
  6. for DNS to propagate.  If it doesn't work, double-check that you have specified a valid
  7. domain name for your cluster, that records have been created in Route53, and that you
  8. can resolve those records from your machine (using `nslookup` or `dig`).
  9.  
  10. ## Other fixes
  11.  
  12. * If you're using a manually created ELB, the auto-scaling groups change, so you will need to reconfigure
  13. your ELBs to include the new auto-scaling group(s).
  14.  
  15. * It is recommended to delete old kubernetes system services that we imported (and replace them with newer versions):

kubectl delete rc -lk8s-app=kube-dns —namespace=kube-system # <= 1.4kubectl delete deployment —namespace=kube-system kube-dns # 1.5

kubectl delete rc -lk8s-app=elasticsearch-logging —namespace=kube-system

kubectl delete rc -lk8s-app=kibana-logging —namespace=kube-system # <= 1.4kubectl delete deployment -lk8s-app=kibana-logging —namespace=kube-system # 1.5

kubectl delete rc -lk8s-app=kubernetes-dashboard —namespace=kube-system # <= 1.4kubectl delete deployment -lk8s-app=kubernetes-dashboard —namespace=kube-system # 1.5

kubectl delete rc -lk8s-app=influxGrafana —namespace=kube-system

kubectl delete deployment -lk8s-app=heapster —namespace=kube-system

  1. ## Delete remaining resources of the old cluster
  2.  
  3. `kops delete cluster ${OLD_NAME}`
  4. >

TYPE NAME IDautoscaling-config kubernetes-minion-group-us-west-2a kubernetes-minion-group-us-west-2aautoscaling-group kubernetes-minion kubernetes-minion-group-us-west-2ainstance kubernetes-master i-67af2ec8```

And once you've confirmed it looks right, run with —yes

You will also need to release the old ElasticIP manually.