Elastic Kubernetes Service (EKS)

Elastic Kubernetes Service (EKS)

LocalStack Pro allows you to use the EKS API to create Kubernetes clusters and easily deploy containerized apps locally.

There are two modes for creating EKS clusters on LocalStack:

  • spinning up an embedded kube cluster in your local Docker engine (preferred, simpler), or
  • using an existing Kubernetes installation you can access from your local machine (defined in $HOME/.kube/config)

Auto-installing an embedded Kubernetes cluster

The default method for creating Kubernetes clusters via the local EKS API is to spin up an embedded k3d kube cluster within Docker. LocalStack handles the download and installation transparently - on most systems the installation is performed automatically, and no customizations should be required.

A new cluster can be created using the following command:

  1. $ awslocal eks create-cluster --name cluster1 --role-arn r1 --resources-vpc-config '{}'

You should then see some Docker containers getting started, e.g.:

  1. $ docker ps
  2. CONTAINER ID   IMAGE                          COMMAND                  CREATED          STATUS          PORTS                                           NAMES
  3. b335f7f089e4   rancher/k3d-proxy:5.0.1-rc.1   "/bin/sh -c nginx-pr…"   1 minute ago   Up 1 minute   0.0.0.0:8081->80/tcp, 0.0.0.0:44959->6443/tcp   k3d-cluster1-serverlb
  4. f05770ec8523   rancher/k3s:v1.21.5-k3s2       "/bin/k3s server --t…"   1 minute ago   Up 1 minute

Once the cluster has been created and initialized, we can determine the server endpoint:

  1. $ awslocal eks describe-cluster --name cluster1
  2. {
  3.     "cluster": {
  4.         "name": "cluster1",
  5.         "status": "ACTIVE",
  6.         "endpoint": "https://localhost.localstack.cloud:4513",
  7.         ...
  8.     }
  9. }

We can then configure the kubectl command line to interact with the new cluster endpoint:

  1. $ awslocal eks update-kubeconfig --name cluster1
  2. Updated context arn:aws:eks:us-east-1:000000000000:cluster/cluster1 in ~/.kube/config
  3. $ kubectl config use-context arn:aws:eks:us-east-1:000000000000:cluster/cluster1
  4. Switched to context "arn:aws:eks:us-east-1:000000000000:cluster/cluster1".
  5. $ kubectl get services
  6. NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
  7. service/kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP   70s

Use images pushed to ECR in EKS

In this section we will, by the use of an example, explore the usage of ECR images inside EKS.

Initial configuration

You can use the configuration variable HOSTNAME_EXTERNAL to modify the return value of the resource URIs for most services, including ECR. By default, ECR will return a repositoryUri starting with localhost, like: localhost:<port>/<repository-name>. If we set the HOSTNAME_EXTERNAL to localhost.localstack.cloud, ECR will return a repositoryUri like localhost.localstack.cloud:<port>/<repository_name>.

Notes

In this section, we will assume localhost.localstack.cloud resolves in your environment and LocalStack is connected to a non-default bridge network. Check the article about DNS rebind protection to learn more. If this domain does not resolve on your host it is also possible not to set HOSTNAME_EXTERNAL, please nevertheless use localhost.localstack.cloud as registry in your pod configuration. LocalStack will take care of the DNS resolution of localhost.localstack.cloud within ECR itself, and you can use the localhost:<port>/<repository_name> Uri for tagging and pushing the image on your host.

If this configuration is correct, you can use your ECR image in EKS like expected.

Deploying a sample application from an ECR image

In order to demonstrate this behavior, take a look at the following small tutorial which leads to the point where the image is correctly pulled. For the sake of this tutorial, we will retag the nginx image to be pushed to ECR using another name, and use it for a pod configuration. First, we create a new repository with a chosen name:

  1. $ awslocal ecr create-repository --repository-name "fancier-nginx"
  2. {
  3.     "repository": {
  4.         "repositoryArn": "arn:aws:ecr:us-east-1:000000000000:repository/fancier-nginx",
  5.         "registryId": "c75fd0e2",
  6.         "repositoryName": "fancier-nginx",
  7.         "repositoryUri": "localhost.localstack.cloud:4510/fancier-nginx",
  8.         "createdAt": "2022-04-13T14:22:47+02:00",
  9.         "imageTagMutability": "MUTABLE",
  10.         "imageScanningConfiguration": {
  11.             "scanOnPush": false
  12.         },
  13.         "encryptionConfiguration": {
  14.             "encryptionType": "AES256"
  15.         }
  16.     }
  17. }

Note: When creating an ECR, a port from the the external service port range is dynamically selected.
Therefore, the port can differ from 4510 used in the samples below. Make sure to use the correct URL / port by using the repositoryUrl of the create-repository request.

Now let us pull the nginx image:

  1. $ docker pull nginx

… tag it to our repository name:

  1. $ docker tag nginx localhost.localstack.cloud:4510/fancier-nginx

… and push it to ECR:

  1. $ docker push localhost.localstack.cloud:4510/fancier-nginx

Now, let us set up the EKS cluster using the image pushed to local ECR.

  1. $ awslocal eks create-cluster --name fancier-cluster --role-arn "r1" --resources-vpc-config "{}"
  2. {
  3.     "cluster": {
  4.         "name": "fancier-cluster",
  5.         "arn": "arn:aws:eks:us-east-1:000000000000:cluster/fancier-cluster",
  6.         "createdAt": "2022-04-13T16:38:24.850000+02:00",
  7.         "roleArn": "r1",
  8.         "resourcesVpcConfig": {},
  9.         "identity": {
  10.             "oidc": {
  11.                 "issuer": "https://localhost.localstack.cloud/eks-oidc"
  12.             }
  13.         },
  14.         "status": "CREATING",
  15.         "clientRequestToken": "cbdf2bb6-fd3b-42b1-afe0-3c70980b5959"
  16.     }
  17. }

Once the cluster status is “ACTIVE”:

  1. awslocal eks describe-cluster --name "fancier-cluster"
  2. {
  3.     "cluster": {
  4.         "name": "fancier-cluster",
  5.         "arn": "arn:aws:eks:us-east-1:000000000000:cluster/fancier-cluster",
  6.         "createdAt": "2022-04-13T17:12:39.738000+02:00",
  7.         "endpoint": "https://localhost.localstack.cloud:4511",
  8.         "roleArn": "r1",
  9.         "resourcesVpcConfig": {},
  10.         "identity": {
  11.             "oidc": {
  12.                 "issuer": "https://localhost.localstack.cloud/eks-oidc"
  13.             }
  14.         },
  15.         "status": "ACTIVE",
  16.         "certificateAuthority": {
  17.             "data": "..."
  18.         },
  19.         "clientRequestToken": "d188f578-b353-416b-b309-5d8c76ecc4e2"
  20.     }
  21. }

… we will configure kubectl:

  1. $ awslocal eks update-kubeconfig --name fancier-cluster && kubectl config use-context arn:aws:eks:us-east-1:000000000000:cluster/fancier-cluster
  2. Added new context arn:aws:eks:us-east-1:000000000000:cluster/fancier-cluster to /home/localstack/.kube/config
  3. Switched to context "arn:aws:eks:us-east-1:000000000000:cluster/fancier-cluster".

… and add a deployment configuration:

  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5.   name: fancier-nginx
  6.   labels:
  7.     app: fancier-nginx
  8. spec:
  9.   replicas: 1
  10.   selector:
  11.     matchLabels:
  12.       app: fancier-nginx
  13.   template:
  14.     metadata:
  15.       labels:
  16.         app: fancier-nginx
  17.     spec:
  18.       containers:
  19.       - name: fancier-nginx
  20.         image: localhost.localstack.cloud:4510/fancier-nginx:latest
  21.         ports:
  22.         - containerPort: 80
  23. EOF

Now, if we describe the pod:

  1. kubectl describe pod fancier-nginx

… we can see, in the events, that the pull from ECR was successful:

  1.   Normal  Pulled     10s   kubelet            Successfully pulled image "localhost.localstack.cloud:4510/fancier-nginx:latest" in 2.412775896s

Configuring an Ingress for your services

In order to make an EKS service externally accessible, we need to create an Ingress configuration that exposes the service on a certain path to the load balancer.

We can create an nginx Kubernetes service for our sample deployment above by applying the following configuration:

  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5.   name: nginx
  6. spec:
  7.   selector:
  8.     app: fancier-nginx
  9.   ports:
  10.   - name: http
  11.     protocol: TCP
  12.     port: 80
  13.     targetPort: 80
  14. EOF

Now use the following ingress configuration to expose the nginx service on path /test123:

  1. $ cat <<EOF | kubectl apply -f -
  2. apiVersion: networking.k8s.io/v1
  3. kind: Ingress
  4. metadata:
  5.   name: nginx
  6.   annotations:
  7.     ingress.kubernetes.io/ssl-redirect: "false"
  8. spec:
  9.   rules:
  10.   - http:
  11.       paths:
  12.       - path: /test123
  13.         pathType: Prefix
  14.         backend:
  15.           service:
  16.             name: nginx
  17.             port:
  18.               number: 80
  19. EOF

We should then be able to send a request to nginx via the load balancer port 8081 from the host:

  1. $ curl http://localhost:8081/test123
  2. <html>
  3. ...
  4. <hr><center>nginx/1.21.6</center>
  5. ...

Note

You can customize the load balancer port by configuring EKS_LOADBALANCER_PORT in your environment.

Using an existing Kubernetes installation

You can also use the EKS API using an existing local Kubernetes installation. This works by mounting the $HOME/.kube/config file into the LocalStack container - e.g., when using docker-compose.yml:

  1. volumes:
  2.   - "${HOME}/.kube/config:/root/.kube/config"

In recent versions of Docker, you can simply enable Kubernetes as an embedded service running inside Docker. See below for a screenshot of the Docker settings for Kubernetes in MacOS (similar configurations apply for Linux/Windows). By default, it is asssumed that Kubernetes API runs on the local TCP port 6443.

Kubernetes in Docker

The example below illustrates how to create an EKS cluster configuration (assuming you have awslocal installed):

  1. $ awslocal eks create-cluster --name cluster1 --role-arn r1 --resources-vpc-config '{}'
  2. {
  3.     "cluster": {
  4.         "name": "cluster1",
  5.         "arn": "arn:aws:eks:eu-central-1:000000000000:cluster/cluster1",
  6.         "createdAt": "Sat, 05 Oct 2019 12:29:26 GMT",
  7.         "endpoint": "https://172.17.0.1:6443",
  8.         "status": "ACTIVE",
  9.         ...
  10.     }
  11. }
  12. $ awslocal eks list-clusters
  13. {
  14.     "clusters": [
  15.         "cluster1"
  16.     ]
  17. }

Simply configure your Kubernetes client (e.g., kubectl or other SDK) to point to the endpoint specified in the create-cluster output above. Depending on whether you’re calling the Kubernetes API from the local machine or from within a Lambda, you may have to use different endpoint URLs (https://localhost:6443 vs https://172.17.0.1:6443).

Mounting directories from host to pod

If you have specific directories which you want to mount from your local dev machine into one of your pods you can do this with two simple steps:

First, make sure to create your cluster with the special tag __k3d_volume_mount__, specifying how you want to mount a volume from your dev machine to the cluster nodes:

  1. $ awslocal eks create-cluster --name cluster1 --role-arn r1 --resources-vpc-config '{}' --tags '{"__k3d_volume_mount__":"/path/on/host:/path/on/node"}'
  2. {
  3.     "cluster": {
  4.         "name": "cluster1",
  5.         "arn": "arn:aws:eks:eu-central-1:000000000000:cluster/cluster1",
  6.         "createdAt": "Sat, 05 Oct 2019 12:29:26 GMT",
  7.         "endpoint": "https://172.17.0.1:6443",
  8.         "status": "ACTIVE",
  9.         "tags": {
  10.             "__k3d_volume_mount__" : "/path/on/host:/path/on/node"
  11.         }
  12.         ...
  13.     }
  14. }

Then, you can create your path with volume mounts as usual, with a configuration similar to this:

  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4.   name: test
  5. spec:
  6.   volumes:
  7.     - name: example-volume
  8.       hostPath:
  9.          path: /path/on/node
  10.   containers:
  11.   - image: alpine:3.12
  12.     command: ["/bin/sh","-c"]
  13.     args:
  14.       - echo "Starting the update command";
  15.         apk update;
  16.         echo "Adding the openssh command";
  17.         apk add openssh;
  18.         echo "openssh completed";
  19.         sleep 240m;
  20.     imagePullPolicy: IfNotPresent
  21.     name: alpine
  22.     volumeMounts:
  23.       - mountPath: "/path/on/pod"
  24.         name: example-volume
  25.   restartPolicy: Always

Last modified June 29, 2022: fix command (2ae75470)