Identity-Aware and HTTP-Aware Policy Enforcement

If you haven’t read the Introduction to Cilium & Hubble yet, we’d encourage you to do that first.

The best way to get help if you get stuck is to ask a question on the Cilium Slack channel. With Cilium contributors across the globe, there is almost always someone available to help.

Setup Cilium

If you have not set up Cilium yet, follow the guide Quick Installation for instructions on how to quickly bootstrap a Kubernetes cluster and install Cilium. If in doubt, pick the minikube route, you will be good to go in less than 5 minutes.

Deploy the Demo Application

Now that we have Cilium deployed and kube-dns operating correctly we can deploy our demo application.

In our Star Wars-inspired example, there are three microservices applications: deathstar, tiefighter, and xwing. The deathstar runs an HTTP webservice on port 80, which is exposed as a Kubernetes Service to load-balance requests to deathstar across two pod replicas. The deathstar service provides landing services to the empire’s spaceships so that they can request a landing port. The tiefighter pod represents a landing-request client service on a typical empire ship and xwing represents a similar service on an alliance ship. They exist so that we can test different security policies for access control to deathstar landing services.

Application Topology for Cilium and Kubernetes

../../_images/cilium_http_gsg.png

The file http-sw-app.yaml contains a Kubernetes Deployment for each of the three services. Each deployment is identified using the Kubernetes labels (org=empire, class=deathstar), (org=empire, class=tiefighter), and (org=alliance, class=xwing). It also includes a deathstar-service, which load-balances traffic to all pods with label (org=empire, class=deathstar).

  1. $ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.12/examples/minikube/http-sw-app.yaml
  2. service/deathstar created
  3. deployment.extensions/deathstar created
  4. pod/tiefighter created
  5. pod/xwing created

Kubernetes will deploy the pods and service in the background. Running kubectl get pods,svc will inform you about the progress of the operation. Each pod will go through several states until it reaches Running at which point the pod is ready.

  1. $ kubectl get pods,svc
  2. NAME                             READY   STATUS    RESTARTS   AGE
  3. pod/deathstar-6fb5694d48-5hmds   1/1     Running   0          107s
  4. pod/deathstar-6fb5694d48-fhf65   1/1     Running   0          107s
  5. pod/tiefighter                   1/1     Running   0          107s
  6. pod/xwing                        1/1     Running   0          107s
  8. NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
  9. service/deathstar    ClusterIP   10.96.110.8   <none>        80/TCP    107s
  10. service/kubernetes   ClusterIP   10.96.0.1     <none>        443/TCP   3m53s

Each pod will be represented in Cilium as an Endpoint. We can invoke the cilium tool inside the Cilium pod to list them:

  1. $ kubectl -n kube-system get pods -l k8s-app=cilium
  2. NAME           READY   STATUS    RESTARTS   AGE
  3. cilium-5ngzd   1/1     Running   0          3m19s
  5. $ kubectl -n kube-system exec cilium-1c2cz -- cilium endpoint list
  6. ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                       IPv6   IPv4         STATUS
  7.            ENFORCEMENT        ENFORCEMENT
  8. 232        Disabled           Disabled          16530      k8s:class=deathstar                                      10.0.0.147   ready
  9.                                                            k8s:io.cilium.k8s.policy.cluster=default
  10.                                                            k8s:io.cilium.k8s.policy.serviceaccount=default
  11.                                                            k8s:io.kubernetes.pod.namespace=default
  12.                                                            k8s:org=empire
  13. 726        Disabled           Disabled          1          reserved:host                                                         ready
  14. 883        Disabled           Disabled          4          reserved:health                                          10.0.0.244   ready
  15. 1634       Disabled           Disabled          51373      k8s:io.cilium.k8s.policy.cluster=default                 10.0.0.118   ready
  16.                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns
  17.                                                            k8s:io.kubernetes.pod.namespace=kube-system
  18.                                                            k8s:k8s-app=kube-dns
  19. 1673       Disabled           Disabled          31028      k8s:class=tiefighter                                     10.0.0.112   ready
  20.                                                            k8s:io.cilium.k8s.policy.cluster=default
  21.                                                            k8s:io.cilium.k8s.policy.serviceaccount=default
  22.                                                            k8s:io.kubernetes.pod.namespace=default
  23.                                                            k8s:org=empire
  24. 2811       Disabled           Disabled          51373      k8s:io.cilium.k8s.policy.cluster=default                 10.0.0.47    ready
  25.                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns
  26.                                                            k8s:io.kubernetes.pod.namespace=kube-system
  27.                                                            k8s:k8s-app=kube-dns
  28. 2843       Disabled           Disabled          16530      k8s:class=deathstar                                      10.0.0.89    ready
  29.                                                            k8s:io.cilium.k8s.policy.cluster=default
  30.                                                            k8s:io.cilium.k8s.policy.serviceaccount=default
  31.                                                            k8s:io.kubernetes.pod.namespace=default
  32.                                                            k8s:org=empire
  33. 3184       Disabled           Disabled          22654      k8s:class=xwing                                          10.0.0.30    ready
  34.                                                            k8s:io.cilium.k8s.policy.cluster=default
  35.                                                            k8s:io.cilium.k8s.policy.serviceaccount=default
  36.                                                            k8s:io.kubernetes.pod.namespace=default
  37.                                                            k8s:org=alliance

Both ingress and egress policy enforcement is still disabled on all of these pods because no network policy has been imported yet which select any of the pods.

Check Current Access

From the perspective of the deathstar service, only the ships with label org=empire are allowed to connect and request landing. Since we have no rules enforced, both xwing and tiefighter will be able to request landing. To test this, use the commands below.

  1. $ kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
  2. Ship landed
  3. $ kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
  4. Ship landed

Apply an L3/L4 Policy

When using Cilium, endpoint IP addresses are irrelevant when defining security policies. Instead, you can use the labels assigned to the pods to define security policies. The policies will be applied to the right pods based on the labels irrespective of where or when it is running within the cluster.

We’ll start with the basic policy restricting deathstar landing requests to only the ships that have label (org=empire). This will not allow any ships that don’t have the org=empire label to even connect with the deathstar service. This is a simple policy that filters only on IP protocol (network layer 3) and TCP protocol (network layer 4), so it is often referred to as an L3/L4 network security policy.

Note: Cilium performs stateful connection tracking, meaning that if policy allows the frontend to reach backend, it will automatically allow all required reply packets that are part of backend replying to frontend within the context of the same TCP/UDP connection.

L4 Policy with Cilium and Kubernetes

../../_images/cilium_http_l3_l4_gsg.png

We can achieve that with the following CiliumNetworkPolicy:

  1. apiVersion: "cilium.io/v2"
  2. kind: CiliumNetworkPolicy
  3. metadata:
  4.   name: "rule1"
  5. spec:
  6.   description: "L3-L4 policy to restrict deathstar access to empire ships only"
  7.   endpointSelector:
  8.     matchLabels:
  9.       org: empire
  10.       class: deathstar
  11.   ingress:
  12.   - fromEndpoints:
  13.     - matchLabels:
  14.         org: empire
  15.     toPorts:
  16.     - ports:
  17.       - port: "80"
  18.         protocol: TCP

CiliumNetworkPolicies match on pod labels using an “endpointSelector” to identify the sources and destinations to which the policy applies. The above policy whitelists traffic sent from any pods with label (org=empire) to deathstar pods with label (org=empire, class=deathstar) on TCP port 80.

To apply this L3/L4 policy, run:

  1. $ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.12/examples/minikube/sw_l3_l4_policy.yaml
  2. ciliumnetworkpolicy.cilium.io/rule1 created

Now if we run the landing requests again, only the tiefighter pods with the label org=empire will succeed. The xwing pods will be blocked!

  1. $ kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
  2. Ship landed

This works as expected. Now the same request run from an xwing pod will fail:

  1. $ kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing

This request will hang, so press Control-C to kill the curl request, or wait for it to time out.

Inspecting the Policy

If we run cilium endpoint list again we will see that the pods with the label org=empire and class=deathstar now have ingress policy enforcement enabled as per the policy above.

  1. $ kubectl -n kube-system exec cilium-1c2cz -- cilium endpoint list
  2. ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                       IPv6   IPv4         STATUS
  3.            ENFORCEMENT        ENFORCEMENT
  4. 232        Enabled            Disabled          16530      k8s:class=deathstar                                      10.0.0.147   ready
  5.                                                            k8s:io.cilium.k8s.policy.cluster=default
  6.                                                            k8s:io.cilium.k8s.policy.serviceaccount=default
  7.                                                            k8s:io.kubernetes.pod.namespace=default
  8.                                                            k8s:org=empire
  9. 726        Disabled           Disabled          1          reserved:host                                                         ready
  10. 883        Disabled           Disabled          4          reserved:health                                          10.0.0.244   ready
  11. 1634       Disabled           Disabled          51373      k8s:io.cilium.k8s.policy.cluster=default                 10.0.0.118   ready
  12.                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns
  13.                                                            k8s:io.kubernetes.pod.namespace=kube-system
  14.                                                            k8s:k8s-app=kube-dns
  15. 1673       Disabled           Disabled          31028      k8s:class=tiefighter                                     10.0.0.112   ready
  16.                                                            k8s:io.cilium.k8s.policy.cluster=default
  17.                                                            k8s:io.cilium.k8s.policy.serviceaccount=default
  18.                                                            k8s:io.kubernetes.pod.namespace=default
  19.                                                            k8s:org=empire
  20. 2811       Disabled           Disabled          51373      k8s:io.cilium.k8s.policy.cluster=default                 10.0.0.47    ready
  21.                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns
  22.                                                            k8s:io.kubernetes.pod.namespace=kube-system
  23.                                                            k8s:k8s-app=kube-dns
  24. 2843       Enabled            Disabled          16530      k8s:class=deathstar                                      10.0.0.89    ready
  25.                                                            k8s:io.cilium.k8s.policy.cluster=default
  26.                                                            k8s:io.cilium.k8s.policy.serviceaccount=default
  27.                                                            k8s:io.kubernetes.pod.namespace=default
  28.                                                            k8s:org=empire
  29. 3184       Disabled           Disabled          22654      k8s:class=xwing                                          10.0.0.30    ready
  30.                                                            k8s:io.cilium.k8s.policy.cluster=default
  31.                                                            k8s:io.cilium.k8s.policy.serviceaccount=default
  32.                                                            k8s:io.kubernetes.pod.namespace=default
  33.                                                            k8s:org=alliance

You can also inspect the policy details via kubectl

  1. $ kubectl get cnp
  2. NAME    AGE
  3. rule1   2m
  5. $ kubectl describe cnp rule1
  6. Name:         rule1
  7. Namespace:    default
  8. Labels:       <none>
  9. Annotations:  <none>
  10. API Version:  cilium.io/v2
  11. Description:  L3-L4 policy to restrict deathstar access to empire ships only
  12. Kind:         CiliumNetworkPolicy
  13. Metadata:
  14.   Creation Timestamp:  2020-06-15T14:06:48Z
  15.   Generation:          1
  16.   Managed Fields:
  17.     API Version:  cilium.io/v2
  18.     Fields Type:  FieldsV1
  19.     fieldsV1:
  20.       f:description:
  21.       f:spec:
  22.         .:
  23.         f:endpointSelector:
  24.           .:
  25.           f:matchLabels:
  26.             .:
  27.             f:class:
  28.             f:org:
  29.         f:ingress:
  30.     Manager:         kubectl
  31.     Operation:       Update
  32.     Time:            2020-06-15T14:06:48Z
  33.   Resource Version:  2914
  34.   Self Link:         /apis/cilium.io/v2/namespaces/default/ciliumnetworkpolicies/rule1
  35.   UID:               eb3a688b-b3aa-495c-b20a-d4f79e7c088d
  36. Spec:
  37.   Endpoint Selector:
  38.     Match Labels:
  39.       Class:  deathstar
  40.       Org:    empire
  41.   Ingress:
  42.     From Endpoints:
  43.       Match Labels:
  44.         Org:  empire
  45.     To Ports:
  46.       Ports:
  47.         Port:      80
  48.         Protocol:  TCP
  49. Events:            <none>

Apply and Test HTTP-aware L7 Policy

In the simple scenario above, it was sufficient to either give tiefighter / xwing full access to deathstar’s API or no access at all. But to provide the strongest security (i.e., enforce least-privilege isolation) between microservices, each service that calls deathstar’s API should be limited to making only the set of HTTP requests it requires for legitimate operation.

For example, consider that the deathstar service exposes some maintenance APIs which should not be called by random empire ships. To see this run:

  1. $ kubectl exec tiefighter -- curl -s -XPUT deathstar.default.svc.cluster.local/v1/exhaust-port
  2. Panic: deathstar exploded
  4. goroutine 1 [running]:
  5. main.HandleGarbage(0x2080c3f50, 0x2, 0x4, 0x425c0, 0x5, 0xa)
  6.         /code/src/github.com/empire/deathstar/
  7.         temp/main.go:9 +0x64
  8. main.main()
  9.         /code/src/github.com/empire/deathstar/
  10.         temp/main.go:5 +0x85

While this is an illustrative example, unauthorized access such as above can have adverse security repercussions.

L7 Policy with Cilium and Kubernetes

../../_images/cilium_http_l3_l4_l7_gsg.png

Cilium is capable of enforcing HTTP-layer (i.e., L7) policies to limit what URLs the tiefighter is allowed to reach. Here is an example policy file that extends our original policy by limiting tiefighter to making only a POST /v1/request-landing API call, but disallowing all other calls (including PUT /v1/exhaust-port).

  1. apiVersion: "cilium.io/v2"
  2. kind: CiliumNetworkPolicy
  3. metadata:
  4.   name: "rule1"
  5. spec:
  6.   description: "L7 policy to restrict access to specific HTTP call"
  7.   endpointSelector:
  8.     matchLabels:
  9.       org: empire
  10.       class: deathstar
  11.   ingress:
  12.   - fromEndpoints:
  13.     - matchLabels:
  14.         org: empire
  15.     toPorts:
  16.     - ports:
  17.       - port: "80"
  18.         protocol: TCP
  19.       rules:
  20.         http:
  21.         - method: "POST"
  22.           path: "/v1/request-landing"

Update the existing rule to apply L7-aware policy to protect deathstar using:

  1. $ kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/v1.12/examples/minikube/sw_l3_l4_l7_policy.yaml
  2. ciliumnetworkpolicy.cilium.io/rule1 configured

We can now re-run the same test as above, but we will see a different outcome:

  1. $ kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
  2. Ship landed

and

  1. $ kubectl exec tiefighter -- curl -s -XPUT deathstar.default.svc.cluster.local/v1/exhaust-port
  2. Access denied

As this rule builds on the identity-aware rule, traffic from pods without the label org=empire will continue to be dropped causing the connection to time out:

  1. $ kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing

As you can see, with Cilium L7 security policies, we are able to permit tiefighter to access only the required API resources on deathstar, thereby implementing a “least privilege” security approach for communication between microservices. Note that path matches the exact url, if for example you want to allow anything under /v1/, you need to use a regular expression:

  1. path: "/v1/.*"

You can observe the L7 policy via kubectl:

  1. $ kubectl describe ciliumnetworkpolicies
  2. Name:         rule1
  3. Namespace:    default
  4. Labels:       <none>
  5. Annotations:  API Version:  cilium.io/v2
  6. Description:  L7 policy to restrict access to specific HTTP call
  7. Kind:         CiliumNetworkPolicy
  8. Metadata:
  9.   Creation Timestamp:  2020-06-15T14:06:48Z
  10.   Generation:          2
  11.   Managed Fields:
  12.     API Version:  cilium.io/v2
  13.     Fields Type:  FieldsV1
  14.     fieldsV1:
  15.       f:description:
  16.       f:metadata:
  17.         f:annotations:
  18.           .:
  19.           f:kubectl.kubernetes.io/last-applied-configuration:
  20.       f:spec:
  21.         .:
  22.         f:endpointSelector:
  23.           .:
  24.           f:matchLabels:
  25.             .:
  26.             f:class:
  27.             f:org:
  28.         f:ingress:
  29.     Manager:         kubectl
  30.     Operation:       Update
  31.     Time:            2020-06-15T14:10:46Z
  32.   Resource Version:  3445
  33.   Self Link:         /apis/cilium.io/v2/namespaces/default/ciliumnetworkpolicies/rule1
  34.   UID:               eb3a688b-b3aa-495c-b20a-d4f79e7c088d
  35. Spec:
  36.   Endpoint Selector:
  37.     Match Labels:
  38.       Class:  deathstar
  39.       Org:    empire
  40.   Ingress:
  41.     From Endpoints:
  42.       Match Labels:
  43.         Org:  empire
  44.     To Ports:
  45.       Ports:
  46.         Port:      80
  47.         Protocol:  TCP
  48.       Rules:
  49.         Http:
  50.           Method:  POST
  51.           Path:    /v1/request-landing
  52. Events:            <none>

and cilium CLI:

  1. $ kubectl -n kube-system exec cilium-qh5l2 -- cilium policy get
  2. [
  3.   {
  4.     "endpointSelector": {
  5.       "matchLabels": {
  6.         "any:class": "deathstar",
  7.         "any:org": "empire",
  8.         "k8s:io.kubernetes.pod.namespace": "default"
  9.       }
  10.     },
  11.     "ingress": [
  12.       {
  13.         "fromEndpoints": [
  14.           {
  15.             "matchLabels": {
  16.               "any:org": "empire",
  17.               "k8s:io.kubernetes.pod.namespace": "default"
  18.             }
  19.           }
  20.         ],
  21.         "toPorts": [
  22.           {
  23.             "ports": [
  24.               {
  25.                 "port": "80",
  26.                 "protocol": "TCP"
  27.               }
  28.             ],
  29.             "rules": {
  30.               "http": [
  31.                 {
  32.                   "path": "/v1/request-landing",
  33.                   "method": "POST"
  34.                 }
  35.               ]
  36.             }
  37.           }
  38.         ]
  39.       }
  40.     ],
  41.     "labels": [
  42.       {
  43.         "key": "io.cilium.k8s.policy.derived-from",
  44.         "value": "CiliumNetworkPolicy",
  45.         "source": "k8s"
  46.       },
  47.       {
  48.         "key": "io.cilium.k8s.policy.name",
  49.         "value": "rule1",
  50.         "source": "k8s"
  51.       },
  52.       {
  53.         "key": "io.cilium.k8s.policy.namespace",
  54.         "value": "default",
  55.         "source": "k8s"
  56.       },
  57.       {
  58.         "key": "io.cilium.k8s.policy.uid",
  59.         "value": "eb3a688b-b3aa-495c-b20a-d4f79e7c088d",
  60.         "source": "k8s"
  61.       }
  62.     ]
  63.   }
  64. ]
  65. Revision: 11

It is also possible to monitor the HTTP requests live by using cilium monitor:

  1. $ kubectl exec -it -n kube-system cilium-kzgdx -- cilium monitor -v --type l7
  2. <- Response http to 0 ([k8s:class=tiefighter k8s:io.cilium.k8s.policy.cluster=default k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.kubernetes.pod.namespace=default k8s:org=empire]) from 2756 ([k8s:io.cilium.k8s.policy.cluster=default k8s:class=deathstar k8s:org=empire k8s:io.kubernetes.pod.namespace=default k8s:io.cilium.k8s.policy.serviceaccount=default]), identity 8876->43854, verdict Forwarded POST http://deathstar.default.svc.cluster.local/v1/request-landing => 200
  3. <- Request http from 0 ([k8s:class=tiefighter k8s:io.cilium.k8s.policy.cluster=default k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.kubernetes.pod.namespace=default k8s:org=empire]) to 2756 ([k8s:io.cilium.k8s.policy.cluster=default k8s:class=deathstar k8s:org=empire k8s:io.kubernetes.pod.namespace=default k8s:io.cilium.k8s.policy.serviceaccount=default]), identity 8876->43854, verdict Denied PUT http://deathstar.default.svc.cluster.local/v1/request-landing => 403

The above output demonstrates a successful response to a POST request followed by a PUT request that is denied by the L7 policy.

We hope you enjoyed the tutorial. Feel free to play more with the setup, read the rest of the documentation, and reach out to us on the Cilium Slack channel with any questions!

Clean-up

  1. $ kubectl delete -f https://raw.githubusercontent.com/cilium/cilium/v1.12/examples/minikube/http-sw-app.yaml
  2. $ kubectl delete cnp rule1