Kubernetes Gateway API

Kuma supports Kubernetes Gateway API for configuring built-in gateway as well as traffic routing using the experimental GAMMA routing spec.

Installation

Kubernetes doesn’t include Gateway API CRDs, install them from the standard channel CRD bundle.

Gateways

  1. Install the counter demo.

    1. kumactl install demo | kubectl apply -f -

  2. Add a Gateway.

    The Gateway resource represents the proxy instance that handles traffic for a set of Gateway API routes.

    Every Gateway refers to a GatewayClass. The GatewayClass represents the class of Gateway, in this case Kuma’s builtin edge gateway, and points to a controller that should manage these Gateways. It can also hold additional configuration.

    For Helm and kumactl installations, a GatewayClass named kuma is automatically installed if the Gateway API CRDs are present.

    If you’ve installed Kuma some other way, you can create your own GatewayClass using the controllerName: gateways.kuma.io/controller:

    1. apiVersion: gateway.networking.k8s.io/v1
    2. kind: GatewayClass
    3. metadata:
    4.   name: kuma
    5. spec:
    6.   controllerName: gateways.kuma.io/controller
    1. apiVersion: gateway.networking.k8s.io/v1
    2. kind: Gateway
    3. metadata:
    4.   name: kuma
    5.   namespace: kuma-demo
    6. spec:
    7.   gatewayClassName: kuma
    8.   listeners:
    9.   - name: proxy
    10.     port: 8080
    11.     protocol: HTTP

    When a user applies a Gateway resource, Kuma automatically creates a Deployment of built-in gateways with a corresponding Service.

    1. kubectl get pods -n kuma-demo
    1. NAME                          READY   STATUS    RESTARTS   AGE
    2. redis-59c9d56fc-6gcbc         2/2     Running   0          2m8s
    3. demo-app-5845d6447b-v7npw     2/2     Running   0          2m8s
    4. kuma-4j6wr-58998b5576-25wl6   1/1     Running   0          30s
    1. kubectl get svc -n kuma-demo
    1. NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
    2. redis        ClusterIP      10.43.223.223   <none>        6379/TCP         3m27s
    3. demo-app     ClusterIP      10.43.216.203   <none>        5000/TCP         3m27s
    4. kuma-pfh4s   LoadBalancer   10.43.122.93    172.20.0.3    8080:30627/TCP   87s

    The Gateway is now accessible using the external address 172.20.0.3:8080.

  3. Add an HTTPRoute.

    HTTPRoute resources contain a set of matching criteria for HTTP requests and upstream Services to route those requests to.

    1. apiVersion: gateway.networking.k8s.io/v1
    2. kind: HTTPRoute
    3. metadata:
    4.   name: echo
    5.   namespace: kuma-demo
    6. spec:
    7.   parentRefs:
    8.   - group: gateway.networking.k8s.io
    9.     kind: Gateway
    10.     name: kuma
    11.     namespace: kuma-demo
    12.   rules:
    13.   - backendRefs:
    14.     - kind: Service
    15.       name: demo-app
    16.       port: 5000
    17.       weight: 1
    18.     matches:
    19.     - path:
    20.         type: PathPrefix
    21.         value: /

    After creating an HTTPRoute, accessing / forwards a request to the demo app:

    1. curl 172.20.0.3:8080/ -i
    1. HTTP/1.1 200 OK
    2. x-powered-by: Express
    3. accept-ranges: bytes
    4. cache-control: public, max-age=0
    5. last-modified: Tue, 20 Oct 2020 17:16:41 GMT
    6. etag: W/"2b91-175470350a8"
    7. content-type: text/html; charset=UTF-8
    8. content-length: 11153
    9. date: Fri, 18 Mar 2022 11:33:29 GMT
    10. x-envoy-upstream-service-time: 2
    11. server: Kuma Gateway
    13. <html>
    14. <head>
    15. ...

TLS termination

Gateway API supports TLS termination by using standard kubernetes.io/tls Secrets.

Here is an example

  1. apiVersion: v1
  2. kind: Secret
  3. metadata:
  4.   name: secret-tls
  5.   namespace: kuma-demo
  6. type: kubernetes.io/tls
  7. data:
  8.   tls.crt: "MIIEOzCCAyO..." # redacted
  9.   tls.key: "MIIEowIBAAKC..." # redacted
  1. apiVersion: gateway.networking.k8s.io/v1
  2. kind: Gateway
  3. metadata:
  4.   name: kuma
  5.   namespace: kuma-demo
  6. spec:
  7.   gatewayClassName: kuma
  8.   listeners:
  9.   - name: proxy
  10.     port: 8080
  11.     hostname: test.kuma.io
  12.     protocol: HTTPS
  13.     tls:
  14.       certificateRefs:
  15.       - name: secret-tls

Under the hood, Kuma CP copies the Secret to kuma-system namespace and converts it to Kuma secret . It tracks all the changes to the secret and deletes it upon deletion of the original secret.

Customization

Gateway API provides the parametersRef field on GatewayClass.spec to provide additional, implementation-specific configuration to Gateways. When using Gateway API with Kuma, you can refer to a MeshGatewayConfig resource:

  1. apiVersion: gateway.networking.k8s.io/v1
  2. kind: GatewayClass
  3. metadata:
  4.   name: kuma
  5. spec:
  6.   controllerName: gateways.kuma.io/controller
  7.   parametersRef:
  8.     kind: MeshGatewayConfig
  9.     group: kuma.io
  10.     name: kuma

This resource has the same structure as the MeshGatewayInstance resource except that the tags field is optional. With a MeshGatewayConfig you can then customize the generated Service and Deployment resources.

Multi-mesh

You can specify a Mesh for Gateway and HTTPRoute resources by setting the kuma.io/mesh annotation Note that HTTPRoutes must also have the annotation to reference a Gateway from a non-default Mesh.

Cross-mesh

Cross-mesh gateways are supported with Gateway API. You’ll just need to create a corresponding GatewayClass pointing to a MeshGatewayConfig that sets crossMesh: true:

  1. apiVersion: gateway.networking.k8s.io/v1
  2. kind: GatewayClass
  3. metadata:
  4.   name: kuma-cross-mesh
  5. spec:
  6.   controllerName: gateways.kuma.io/controller
  7.   parametersRef:
  8.     group: kuma.io
  9.     kind: MeshGatewayConfig
  10.     name: default-cross-mesh
  11. ---
  12. apiVersion: kuma.io/v1alpha1
  13. kind: MeshGatewayConfig
  14. metadata:
  15.   name: default-cross-mesh
  16. spec:
  17.   crossMesh: true

and then reference it in your Gateway:

  1. apiVersion: gateway.networking.k8s.io/v1
  2. kind: Gateway
  3. metadata:
  4.   name: kuma
  5.   namespace: default
  6. spec:
  7.   gatewayClassName: kuma-cross-mesh
  8.   listeners:
  9.   - name: proxy
  10.     port: 8080
  11.     protocol: HTTP

Multi-zone Deployments

The Gateway API supports multi-zone deployments, but with some limitations:

  • Gateway API resources like Gateway, ReferenceGrant, and HTTPRoute must be created in non-global zones.

  • Only services deployed within the same Kubernetes cluster, such as the HTTPRoute, can be referenced via backendRef.

    Important Note: This limitation exist because, Kuma currently only allows referencing as backendRefs Kubernetes Services.

    This is a temporary limitation. We’re actively working on extending backendRef to support Kuma’s MeshServices. Once this feature is complete, you’ll be able to reference services across different clusters within your mesh.

    To better visualize this limitation here’s an example scenario that describes how you could configure multi-zone deployments with the Gateway API. In this example, you have the following resources:

    • Two zones (zone-1 and zone-2) in separate Kubernetes clusters

    • Gateway with listener on port 8080 deployed in zone-1

    • Two services:

      • A service named backend deployed in each zone

      • A service named db deployed only in zone-2

    2.  flowchart TD
    3.      subgraph c2["k8s-cluster-2"]
    4.          subgraph z2["zone-2"]
    5.              subgraph c1z2s1["Service"]
    6.                  b2(backend)
    7.              end
    8.              subgraph c1z2s2["Service"]
    9.                  db(db)
    10.              end
    11.          end
    12.      end
    13.      subgraph c1["k8s-cluster-1"]
    14.          subgraph z1["zone-1"]
    15.              subgraph Gateway
    16.                  listener(:8080)
    17.              end
    18.              subgraph Service
    19.                  b1(backend)
    20.              end
    21.          end
    22.      end

    If you deploy multi-zone with Gateway API, the following will occur:

    • If you create an HTTPRoute with a backendRef targeting the backend service in k8s-cluster-1, it will only route traffic to the backend service in k8s-cluster-1.
    2.  flowchart TD
    3.      subgraph c2["k8s-cluster-2"]
    4.          subgraph z2["zone-2"]
    5.              subgraph c1z2s1["Service"]
    6.                  backend2(backend)
    7.              end
    8.              subgraph c1z2s2["Service"]
    9.                  db(db)
    10.              end
    11.          end
    12.      end
    13.      subgraph c1["k8s-cluster-1"]
    14.          subgraph z1["zone-1"]
    15.              subgraph Service
    16.                  backend1(backend)
    17.              end
    18.              subgraph Gateway
    19.                  listener(:8080)
    20.              end
    21.              subgraph HTTPRoute
    22.                  route1(/)
    23.              end
    24.              route1--"❌"-->backend2
    25.              linkStyle 0 stroke:red,color:red,stroke-dasharray: 5 5;
    26.              route1-->backend1
    27.              listener-->route1
    28.          end
    29.      end
    • Similarly, if you create an HTTPRoute with a backendRef pointing to the db service in k8s-cluster-1, it will result in a HTTPRoute with a ResolvedRefs status condition of BackendNotFound because service db is not present in k8s-cluster-1.
    2.  flowchart TD
    3.      subgraph c2["k8s-cluster-2"]
    4.          subgraph z2["zone-2"]
    5.              subgraph c1z2s1["Service"]
    6.                  backend2(backend)
    7.              end
    8.              subgraph c1z2s2["Service"]
    9.                  db(db)
    10.              end
    11.          end
    12.      end
    13.      subgraph c1["k8s-cluster-1"]
    14.          subgraph z1["zone-1"]
    15.              subgraph Gateway
    16.                  listener(:8080)
    17.              end
    18.              subgraph HTTPRoute
    19.                  route1(/)
    20.              end
    21.              subgraph Service
    22.                  backend1(backend)
    23.              end
    24.              route1--"❌"-->db
    25.              linkStyle 0 stroke:red,color:red,stroke-dasharray: 5 5;
    26.              listener-->route1
    27.          end
    28.      end

Service to service routing

Kuma also supports routing between services with HTTPRoute in conformance with the GAMMA specifications.

GAMMA is a Gateway API subproject focused on mesh implementations of Gateway API and extending the Gateway API resources to mesh use cases.

The key feature of HTTPRoute for mesh routing is specifying a Kubernetes Service as the parentRef, as opposed to a Gateway. All requests to this Service are then filtered and routed as specified in the HTTPRoute.

  1. apiVersion: gateway.networking.k8s.io/v1
  2. kind: HTTPRoute
  3. metadata:
  4.   name: canary-demo-app
  5.   namespace: kuma-demo
  6. spec:
  7.   parentRefs:
  8.   - name: demo-app
  9.     port: 5000
  10.     kind: Service
  11.   rules:
  12.   - backendRefs:
  13.     - name: demo-app-v1
  14.       port: 5000
  15.     - name: demo-app-v2
  16.       port: 5000

The namespace of the HTTPRoute is key. If the route’s namespace and the parentRef’s namespace are identical, Kuma applies the route to requests from all workloads. If the route’s namespace differs from its parentRef’s namespace, the HTTPRoute applies only to requests from workloads in the route’s namespace.

Remember to tag your Service ports with appProtocol: http to use them in an HTTPRoute!

Because of how Kuma maps resources at the moment, the combination of the HTTPRoutes name and namespace and the parent Service name and namespace must be no more than 249 characters.

How it works

Kuma includes controllers that reconcile Gateway API CRDs and convert them into the corresponding Kuma CRDs. This is why in the GUI, Kuma MeshGateways/MeshHTTPRoutes/MeshTCPRoutes are visible and not Kubernetes Gateway API resources.

Kubernetes Gateway API resources serve as the source of truth for Kuma gateways and routes. Any edits to the corresponding Kuma resources are overwritten.