Getting Started with kOps on AWS

Make sure you have installed kOps and installed kubectl.

Setup your environment

AWS

In order to correctly prepare your AWS account for kops, we require you to install the AWS CLI tools, and have API credentials for an account that has the permissions to create a new IAM account for kops later in the guide.

Once you’ve installed the AWS CLI tools and have correctly setup your system to use the official AWS methods of registering security credentials as defined here we’ll be ready to run kops, as it uses the Go AWS SDK.

Setup IAM user

In order to build clusters within AWS we’ll create a dedicated IAM user for kops. This user requires API credentials in order to use kops. Create the user, and credentials, using the AWS console.

The kops user will require the following IAM permissions to function properly:

  1. AmazonEC2FullAccess
  2. AmazonRoute53FullAccess
  3. AmazonS3FullAccess
  4. IAMFullAccess
  5. AmazonVPCFullAccess
  6. AmazonSQSFullAccess
  7. AmazonEventBridgeFullAccess

You can create the kOps IAM user from the command line using the following:

  1. aws iam create-group --group-name kops
  3. aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonEC2FullAccess --group-name kops
  4. aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonRoute53FullAccess --group-name kops
  5. aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess --group-name kops
  6. aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/IAMFullAccess --group-name kops
  7. aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonVPCFullAccess --group-name kops
  8. aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonSQSFullAccess --group-name kops
  9. aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonEventBridgeFullAccess --group-name kops
  11. aws iam create-user --user-name kops
  13. aws iam add-user-to-group --user-name kops --group-name kops
  15. aws iam create-access-key --user-name kops

You should record the SecretAccessKey and AccessKeyID in the returned JSON output, and then use them below:

  1. # configure the aws client to use your new IAM user
  2. aws configure           # Use your new access and secret key here
  3. aws iam list-users      # you should see a list of all your IAM users here
  5. # Because "aws configure" doesn't export these vars for kops to use, we export them now
  6. export AWS_ACCESS_KEY_ID=$(aws configure get aws_access_key_id)
  7. export AWS_SECRET_ACCESS_KEY=$(aws configure get aws_secret_access_key)

Configure DNS

In order to build a Kubernetes cluster with kops, we need to prepare somewhere to build the required DNS records. There are three scenarios below and you should choose the one that most closely matches your AWS situation.

Note: if you want to use gossip-based DNS, you can skip this section.

Scenario 1a: A Domain purchased/hosted via AWS

If you bought your domain with AWS, then you should already have a hosted zone in Route53. If you plan to use this domain then no more work is needed.

In this example you own example.com and your records for Kubernetes would look like etcd-us-east-1c.internal.clustername.example.com

Scenario 1b: A subdomain under a domain purchased/hosted via AWS

In this scenario you want to contain all kubernetes records under a subdomain of a domain you host in Route53. This requires creating a second hosted zone in route53, and then setting up route delegation to the new zone.

In this example you own example.com and your records for Kubernetes would look like etcd-us-east-1c.internal.clustername.subdomain.example.com

This is copying the NS servers of your SUBDOMAIN up to the PARENT domain in Route53. To do this you should:

  • Create the subdomain, and note your SUBDOMAIN name servers (If you have already done this you can also get the values)
  1. # Note: This example assumes you have jq installed locally.
  2. ID=$(uuidgen) && aws route53 create-hosted-zone --name subdomain.example.com --caller-reference $ID | \
  3.     jq .DelegationSet.NameServers
  • Note your PARENT hosted zone id
  1. # Note: This example assumes you have jq installed locally.
  2. aws route53 list-hosted-zones | jq '.HostedZones[] | select(.Name=="example.com.") | .Id'
  • Create a new JSON file with your values (subdomain.json)

Note: The NS values here are for the SUBDOMAIN

  1. {
  2.   "Comment": "Create a subdomain NS record in the parent domain",
  3.   "Changes": [
  4.     {
  5.       "Action": "CREATE",
  6.       "ResourceRecordSet": {
  7.         "Name": "subdomain.example.com",
  8.         "Type": "NS",
  9.         "TTL": 300,
  10.         "ResourceRecords": [
  11.           {
  12.             "Value": "ns-1.<example-aws-dns>-1.co.uk"
  13.           },
  14.           {
  15.             "Value": "ns-2.<example-aws-dns>-2.org"
  16.           },
  17.           {
  18.             "Value": "ns-3.<example-aws-dns>-3.com"
  19.           },
  20.           {
  21.             "Value": "ns-4.<example-aws-dns>-4.net"
  22.           }
  23.         ]
  24.       }
  25.     }
  26.   ]
  27. }
  • Apply the SUBDOMAIN NS records to the PARENT hosted zone.
  1. aws route53 change-resource-record-sets \
  2.  --hosted-zone-id <parent-zone-id> \
  3.  --change-batch file://subdomain.json

Now traffic to *.subdomain.example.com will be routed to the correct subdomain hosted zone in Route53.

Scenario 2: Setting up Route53 for a domain purchased with another registrar

If you bought your domain elsewhere, and would like to dedicate the entire domain to AWS you should follow the guide here

Scenario 3: Subdomain for clusters in route53, leaving the domain at another registrar

If you bought your domain elsewhere, but only want to use a subdomain in AWS Route53 you must modify your registrar’s NS (NameServer) records. We’ll create a hosted zone in Route53, and then migrate the subdomain’s NS records to your other registrar.

You might need to grab jq for some of these instructions.

  • Create the subdomain, and note your name servers (If you have already done this you can also get the values)
  1. ID=$(uuidgen) && aws route53 create-hosted-zone --name subdomain.example.com --caller-reference $ID | jq .DelegationSet.NameServers

  • You will now go to your registrar’s page and log in. You will need to create a new SUBDOMAIN, and use the 4 NS records received from the above command for the new SUBDOMAIN. This MUST be done in order to use your cluster. Do NOT change your top level NS record, or you might take your site offline.

  • Information on adding NS records with Godaddy.com

  • Information on adding NS records with Google Cloud Platform

Using Public/Private DNS (kOps 1.5+)

By default the assumption is that NS records are publicly available. If you require private DNS records you should modify the commands we run later in this guide to include:

  1. kops create cluster --dns private $NAME

If you have a mix of public and private zones, you will also need to include the --dns-zone argument with the hosted zone id you wish to deploy in:

  1. kops create cluster --dns private --dns-zone ZABCDEFG $NAME

Testing your DNS setup

This section is not required if a gossip-based cluster is created.

You should now be able to dig your domain (or subdomain) and see the AWS Name Servers on the other end.

  1. dig ns subdomain.example.com

Should return something similar to:

  1. ;; ANSWER SECTION:
  2. subdomain.example.com.        172800  IN  NS  ns-1.<example-aws-dns>-1.net.
  3. subdomain.example.com.        172800  IN  NS  ns-2.<example-aws-dns>-2.org.
  4. subdomain.example.com.        172800  IN  NS  ns-3.<example-aws-dns>-3.com.
  5. subdomain.example.com.        172800  IN  NS  ns-4.<example-aws-dns>-4.co.uk.

This is a critical component when setting up clusters. If you are experiencing problems with the Kubernetes API not coming up, chances are something is wrong with the cluster’s DNS.

Please DO NOT MOVE ON until you have validated your NS records! This is not required if a gossip-based cluster is created.

Cluster State store

In order to store the state of your cluster, and the representation of your cluster, we need to create a dedicated S3 bucket for kops to use. This bucket will become the source of truth for our cluster configuration. In this guide we’ll call this bucket example-com-state-store, but you should add a custom prefix as bucket names need to be unique.

We recommend keeping the creation of this bucket confined to us-east-1, otherwise more work will be required.

  1. aws s3api create-bucket \
  2.     --bucket prefix-example-com-state-store \
  3.     --region us-east-1

Note: S3 requires --create-bucket-configuration LocationConstraint=<region> for regions other than us-east-1.

Note: We STRONGLY recommend versioning your S3 bucket in case you ever need to revert or recover a previous state store.

  1. aws s3api put-bucket-versioning --bucket prefix-example-com-state-store  --versioning-configuration Status=Enabled

Cluster OIDC store

In order for ServiceAccounts to use external permissions (aka IAM Roles for ServiceAccounts), you also need a bucket for hosting the OIDC documents. While you can reuse the bucket above if you grant it a public ACL, we do recommend a separate bucket for these files.

The ACL must be public so that the AWS STS service can access them.

  1. aws s3api create-bucket \
  2.     --bucket prefix-example-com-oidc-store \
  3.     --region us-east-1 \
  4.     --object-ownership BucketOwnerPreferred
  5. aws s3api put-public-access-block \
  6.     --bucket prefix-example-com-oidc-store \
  7.     --public-access-block-configuration BlockPublicAcls=false,IgnorePublicAcls=false,BlockPublicPolicy=false,RestrictPublicBuckets=false
  8. aws s3api put-bucket-acl \
  9.     --bucket prefix-example-com-oidc-store \
  10.     --acl public-read

Information regarding cluster state store location must be set when using kops cli. See state store for further information.

Using S3 default bucket encryption

kops supports default bucket encryption to encrypt its state in an S3 bucket. This way, the default server side encryption set for your bucket will be used for the kOps state too. You may want to use this AWS feature, e.g., for easily encrypting every written object by default or when you need to use specific encryption keys (KMS, CMK) for compliance reasons.

If your S3 bucket has a default encryption set up, kOps will use it:

  1. aws s3api put-bucket-encryption --bucket prefix-example-com-state-store --server-side-encryption-configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'

If the default encryption is not set or it cannot be checked, kOps will resort to using server-side AES256 bucket encryption with Amazon S3-Managed Encryption Keys (SSE-S3).

Sharing an S3 bucket across multiple accounts

It is possible to use a single S3 bucket for storing kOps state for clusters located in different accounts by using cross-account bucket policies.

kOps will be able to use buckets configured with cross-account policies by default.

In this case you may want to override the object ACLs which kOps places on the state files, as default AWS ACLs will make it possible for an account that has delegated access to write files that the bucket owner cannot read.

To do this you should set the environment variable KOPS_STATE_S3_ACL to the preferred object ACL, for example: bucket-owner-full-control.

For available canned ACLs please consult Amazon’s S3 documentation.

Creating your first cluster

Prepare local environment

We’re ready to start creating our first cluster! Let’s first set up a few environment variables to make the process easier.

  1. export NAME=myfirstcluster.example.com
  2. export KOPS_STATE_STORE=s3://prefix-example-com-state-store

For a gossip-based cluster, make sure the name ends with k8s.local. For example:

  1. export NAME=myfirstcluster.k8s.local
  2. export KOPS_STATE_STORE=s3://prefix-example-com-state-store

Note: You don’t have to use environmental variables here. You can always define the values using the –name and –state flags later.

Create cluster configuration

We will need to note which availability zones are available to us. In this example we will be deploying our cluster to the us-west-2 region.

  1. aws ec2 describe-availability-zones --region us-west-2

Below is a create cluster command. We’ll use the most basic example possible, with more verbose examples in high availability. The below command will generate a cluster configuration, but will not start building it. Make sure you have generated an SSH key pair before creating your cluster.

  1. kops create cluster \
  2.     --name=${NAME} \
  3.     --cloud=aws \
  4.     --zones=us-west-2a \
  5.     --discovery-store=s3://prefix-example-com-oidc-store/${NAME}/discovery

All instances created by kops will be built within ASG (Auto Scaling Groups), which means each instance will be automatically monitored and rebuilt by AWS if it suffers any failure.

Customize Cluster Configuration

Now we have a cluster configuration, we can look at every aspect that defines our cluster by editing the description.

  1. kops edit cluster --name ${NAME}

This opens your editor (as defined by $EDITOR) and allows you to edit the configuration. The configuration is loaded from the S3 bucket we created earlier, and automatically updated when we save and exit the editor.

We’ll leave everything set to the defaults for now, but the rest of kops documentation covers additional settings and configuration you can enable.

Build the Cluster

Now we take the final step of actually building the cluster. This’ll take a while. Once it finishes you’ll have to wait longer while the booted instances finish downloading Kubernetes components and reach a “ready” state.

  1. kops update cluster --name ${NAME} --yes --admin

Use the Cluster

Remember when you installed kubectl earlier? The configuration for your cluster was automatically generated and written to ~/.kube/config for you!

A simple Kubernetes API call can be used to check if the API is online and listening. Let’s use kubectl to check the nodes.

  1. kubectl get nodes

You will see a list of nodes that should match the --zones flag defined earlier. This is a great sign that your Kubernetes cluster is online and working.

kops also ships with a handy validation tool that can be ran to ensure your cluster is working as expected.

  1. kops validate cluster --wait 10m

You can look at all system components with the following command.

  1. kubectl -n kube-system get po

Delete the Cluster

Running a Kubernetes cluster within AWS obviously costs money, and so you may want to delete your cluster if you are finished running experiments.

You can preview all of the AWS resources that will be destroyed when the cluster is deleted by issuing the following command.

  1. kops delete cluster --name ${NAME}

When you are sure you want to delete your cluster, issue the delete command with the --yes flag. Note that this command is very destructive, and will delete your cluster and everything contained within it!

  1. kops delete cluster --name ${NAME} --yes

Next steps

Now that you have a working kOps cluster, read through the recommendations for production setups guide

Feedback

There’s an incredible team behind kOps and we encourage you to reach out to the community on the Kubernetes Slack (http://slack.k8s.io/). Bring your questions, comments, and requests and meet the people behind the project!

AWS Trademark used with limited permission under the AWS Trademark Guidelines

Kubernetes Logo used with permission under the Kubernetes Branding Guidelines