GEP-724: Refresh Route-Gateway Binding

  • Issue: #724
  • Status: Standard

TLDR

This GEP proposes changes to Route-Gateway binding that will result in Routes attaching to Gateways with direct references. When supporting Routes in multiple namespaces, Gateways will need to specify the namespaces they trust Routes in. These changes will slightly simplify the Route-Gateway relationship and make way for the future addition of Route inclusion (Routes including other Routes).

Goals

Refactor cross-namespace Route-Gateway binding to:

  • Be more consistent with cross-namespace references from Routes
  • Provide a clear path to enable Route inclusion (Routes including Routes).
  • Simplify user experience based on initial feedback.
  • Enable other kinds of Route parents in addition to Gateway, this could include:
    • Routes (as part of one potential approach to Route inclusion)
    • Custom Gateway resources
    • Mesh resources

Out of scope

  • Defining how Route inclusion will work.

Existing Approach

The existing API already supports cross-namespace references. Gateways configure the following:

  • A Route label selector
  • Namespaces: Same, Selector, or All

Routes then have three options as far as which Gateways can bind to them:

  • Same namespace (default)
  • From a list of Gateways
  • All

Although this enables a great deal of flexibility, it can lead to confusion. For example, 2 separate label selectors from Gateway can be challenging for users to compute. It requires users to do to label selector lookups and then compute the intersection of that result. Additionally, the default behavior of selecting all Routes in the same namespace makes it easy to accidentally expose applications (see #515).

Proposed Changes

Although the API changes proposed here are relatively small, this represents a larger conceptual change. Instead of Gateways selecting Routes, Routes would directly reference the Gateways they wanted to attach to. This pattern was already possible with the existing API, but not clearly documented.

One of the key concepts in the cross-namespace references from Routes GEP was that of a handshake for references that cross namespace boundaries. A key part of that handshake was that one direction included a direct reference, while the other direction provided a way to denote trust for a set of Namespaces and kind of resources.

It seems to make sense to carry that same underlying principle through to the Route - Gateway relationship. Given that each Gateway is likely to support many Routes, it would not be practical to support direct references from Gateways to Routes. Instead, it is simpler if Routes directly reference the Gateways they want to attach to. Gateways can then specify the namespaces they trust Routes to attach from.

Routes directly reference Gateway

In the following example, the lb Gateway indicates that it trusts Routes from the foo Namespace, and the HTTPRoute in that namespace attaches directly to the Gateway.

  1. kind: Gateway
  2. metadata:
  3.   name: lb
  4.   namespace: infra
  5. spec:
  6.   listeners:
  7.   - name: foo
  8.     hostname: foo.com
  9.     port: 80
  10.     routes:
  11.       namespaces:
  12.         from: Selector
  13.         selector:
  14.           kubernetes.io/metadata.name: foo
  15. ---
  16. kind: HTTPRoute
  17. metadata:
  18.   name: foo
  19.   namespace: foo
  20. spec:
  21.   attachTo:
  22.   - kind: Gateway
  23.     namespace: infra
  24.     name: lb
  25.     sectionName: foo
  26.   rules:
  27.   - name: abc
  28.     matches:
  29.     - path: /bar

Rationale

1. Remove Complexity While it is Still Possible

A goal of v1alpha2 is to make any breaking changes we think we’ll need to for the lifetime of the API. After this release, we plan to provide a fully convertible, and hopefully also fully backwards compatible, API. If we really do need more advanced functionality in the future, it will be fairly straightforward to add in a future release. On the other hand, it will be near impossible to remove this complexity post-v1alpha2 if we were to leave it as is.

2. Route Selection Added Confusion and Did Not Enhance Security

Although it was easy to look at the selector from Gateway -> Route as providing Gateway admins some form of control over the Routes attached to their Gateway, it was nothing but security theater. Route owners still had ultimate control by deciding how their Routes were labeled. This also made it difficult to document how Gateway and Route owners should interact. One possible explanation was that Gateway owners should provide Route owners with a set of labels that they should add to their Route to bind to a given Gateway. At that point, we were still ultimately relying on Route owners to attach a Route to a Gateway, just making it a rather clunky process.

It should be noted that this proposal does still retain the ability for Gateways to restrict the namespaces they trust Routes to attach from. This was the only real control Gateway admins had before this proposed change.

3. The Existing Defaults Were Too Permissive

One of the common complaints about the existing API was that the defaults made it far too easy to accidentally expose a Route. By default, a Gateway would be bound to all Routes in the same namespace. Although that made the getting started guide simple, it would inevitably lead to some unfortunate mistakes in the future. As we’ve already learned with Kubernetes, it’s very difficult to recover from insecure defaults. Instead, it’s much safer to start with more explicit configuration that demonstrates clear intent to connect resources.

4. We Need to Support non-Gateway Route Parents

With the expansion of this API, it’s clear that a Route may have non-Gateway parents. This may be other Routes, mesh implementations, or custom Gateways. Although none of these concepts are well specified at this point, making this change now will give us more flexibility in the future.

5. Users Want Control over the Gateways Their Routes Are Attached To

Initial feedback we’ve received has shown that users want to have very clear control over the Gateways their Routes are attached to. Even in the case of Gateway replacement, many Route owners would prefer to be involved in the process.

As we get more feedback and usage of the API, we may identify more users that are interested in the more advanced capabilities that some form of selection may enable, but at this point it’s clear that a large portion of users value an explicit way to attach a Route to a Gateway.

6. We Need to Maintain a Handshake Between Gateways and Routes

Of course we do still need a handshake that will enable cross-namespace references between Routes and Gateways. This proposal leaves in the core capabilities of the v1alpha1 API for this. Gateways can specify the namespaces they trust Routes to bind from, and Routes directly reference the Gateways they want to attach to. This is largely similar to the ReferenceGrant model proposed for Route->Service references, but is embedded within the Route and Gateway resources. The alternatives below explore what this could look like with ReferenceGrant.

API Changes

The proposed changes here can be summarized as:

  • Remove Route selector from the RouteBindingSelector in Gateway listeners.
  • Replace Route kind and group with optional list of accepted kinds and groups in RouteBindingSelector.
  • Rename RouteBindingSelector to ListenerRoutes.
  • Replace the 3 options from Route -> Gateway (All, FromList, SameNamespace) with a reference list that supports arbitrary kinds.
  • Add a name to Gateway listeners.
  • Restructure listener status to include name, routeRefs, and supportedKinds fields.

Gateway Spec

In Gateway spec, the only change involves removing the Route selector field. Everything else remains the same.

Removed

The following fields would be removed from RouteBindingSelector:

  1.   // Selector specifies a set of route labels used for selecting
  2.   // routes to associate with the Gateway. If this Selector is defined,
  3.   // only routes matching the Selector are associated with the Gateway.
  4.   // An empty Selector matches all routes.
  5.   //
  6.   // Support: Core
  7.   //
  8.   // +optional
  9.   Selector *metav1.LabelSelector `json:"selector,omitempty"`
  10.   // Group is the group of the route resource to select. Omitting the value or specifying
  11.   // the empty string indicates the gateway.networking.k8s.io API group.
  12.   // For example, use the following to select an HTTPRoute:
  13.   //
  14.   // routes:
  15.   //   kind: HTTPRoute
  16.   //
  17.   // Otherwise, if an alternative API group is desired, specify the desired
  18.   // group:
  19.   //
  20.   // routes:
  21.   //   group: acme.io
  22.   //   kind: FooRoute
  23.   //
  24.   // Support: Core
  25.   //
  26.   // +optional
  27.   // +kubebuilder:default=gateway.networking.k8s.io
  28.   // +kubebuilder:validation:MinLength=1
  29.   // +kubebuilder:validation:MaxLength=253
  30.   Group *string `json:"group,omitempty"`
  31.   // Kind is the kind of the route resource to select.
  32.   //
  33.   // Kind MUST correspond to kinds of routes that are compatible with the
  34.   // application protocol specified in the Listener's Protocol field.
  35.   //
  36.   // If an implementation does not support or recognize this
  37.   // resource type, it SHOULD set the "ResolvedRefs" condition to false for
  38.   // this listener with the "InvalidRoutesRef" reason.
  39.   //
  40.   // Support: Core
  41.   Kind string `json:"kind"`

Added

Note: The ListMapKey annotation for listeners would also have to change to name for this.

  1. type Listener struct {
  2.     // Name is the name of the Listener. If more than one Listener is present
  3.     // each Listener MUST specify a name. The names of Listeners MUST be unique
  4.     // within a Gateway.
  5.     //
  6.     // Support: Core
  7.     //
  8.     // +kubebuilder:validation:MinLength=1
  9.     // +kubebuilder:validation:MaxLength=253
  10.     // +optional
  11.     Name *string `json:"name,omitempty"`
  12.     // ...
  13. }

The RouteBindingSelector struct would be renamed to ListenerRoutes, and a Kinds field would be added. Note that the Selector, Group, and Kind field would be removed from this struct as described above.

  1. type ListenerRoutes struct {
  2.   // ...
  3.   // Kinds specifies the groups and kinds of Routes that are allowed to bind to
  4.   // this Gateway listener. When unspecified or empty, the only limitation on
  5.   // the kinds of Routes supported is the Listener protocol. Kind MUST
  6.   // correspond to kinds of Routes that are compatible with the application
  7.   // protocol specified in the Listener's Protocol field. If an implementation
  8.   // does not support or recognize this resource type, it SHOULD set the
  9.   // "ResolvedRefs" condition to false for this listener with the
  10.   // "InvalidRoutesRef" reason.
  11.   //
  12.   // Support: Core
  13.   //
  14.   // +optional
  15.   // +kubebuilder:validation:MaxItems=10
  16.   Kinds []RouteGroupKind `json:"kinds,omitempty"`
  17. }
  19. type RouteGroupKind struct {
  20.   // Group is the group of the Route.
  21.   //
  22.   // Support: Core
  23.   //
  24.   // +optional
  25.   // +kubebuilder:default=gateway.networking.k8s.io
  26.   // +kubebuilder:validation:MaxLength=253
  27.   Group *string `json:"group,omitempty"`
  28.   // Kind is the kind of the Route.
  29.   //
  30.   // Support: Core
  31.   //
  32.   // +kubebuilder:validation:MinLength=1
  33.   // +kubebuilder:validation:MaxLength=253
  34.   Kind string `json:"kind"`
  35. }

Gateway Status

The most significant addition to the Gateway resource is in status. It may be helpful to share a sample of what the YAML would look like:

  1. status:
  2.   listeners:
  3.   - name: foo
  4.     supportedKinds:
  5.     - group: gateway.networking.k8s.io
  6.       kind: HTTPRoute
  7.     attachedRoutes: 1
  8.     conditions:
  9.     - ...

The key changes here all involve Listener status:

  • Replace the port, protocol, and hostname field with name to take advantage of the new Listener name concept.
  • Add a new supportedKinds field. This will be most useful when the corresponding field in the spec is left blank or when a user specifies kinds that a controller does not support.

Note: The ListMapKey annotation for listener status would also have to change to name for this.

  1. // ListenerStatus is the status associated with a Listener.
  2. type ListenerStatus struct {
  3.     // Name is the name of the Listener.
  4.     //
  5.     // +kubebuilder:validation:MinLength=1
  6.     // +kubebuilder:validation:MaxLength=253
  7.     // +optional
  8.     Name *string `json:"name,omitempty"`
  10.     // SupportedKinds is the list indicating the Kinds supported by this
  11.     // listener. When this is not specified on the Listener, this MUST represent
  12.     // the kinds an implementation supports for the specified protocol. When
  13.     // there are kinds specified on the Listener, this MUST represent the
  14.     // intersection of those kinds and the kinds supported by the implementation
  15.     // for the specified protocol.
  16.     //
  17.     // +kubebuilder:validation:MaxItems=10
  18.     // +optional
  19.     SupportedKinds []RouteGroupKind `json:"supportedKinds,omitempty"`
  21.     // AttachedRoutes represents the total number of Routes that have been
  22.     // successfully attached to this Listener.
  23.     AttachedRoutes int32 `json:"attachedRoutes"`
  25.     // Conditions...
  26. }

Routes

On Routes, we remove the RouteGateways struct and replace it with a list of parent references to attach to.

Removed

From Route Specs:

  1.   // Gateways defines which Gateways can use this Route.
  2.   //
  3.   // +optional
  4.   // +kubebuilder:default={allow: "SameNamespace"}
  5.   Gateways *RouteGateways `json:"gateways,omitempty"`

And the structs that references:

  1. // RouteGateways defines which Gateways will be able to use a route. If this
  2. // field results in preventing the selection of a Route by a Gateway, an
  3. // "Admitted" condition with a status of false must be set for the Gateway on
  4. // that Route.
  5. type RouteGateways struct {
  6.   // Allow indicates which Gateways will be allowed to use this route.
  7.   // Possible values are:
  8.   // * All: Gateways in any namespace can use this route.
  9.   // * FromList: Only Gateways specified in GatewayRefs may use this route.
  10.   // * SameNamespace: Only Gateways in the same namespace may use this route.
  11.   //
  12.   // +optional
  13.   // +kubebuilder:validation:Enum=All;FromList;SameNamespace
  14.   // +kubebuilder:default=SameNamespace
  15.   Allow *GatewayAllowType `json:"allow,omitempty"`
  17.   // GatewayRefs must be specified when Allow is set to "FromList". In that
  18.   // case, only Gateways referenced in this list will be allowed to use this
  19.   // route. This field is ignored for other values of "Allow".
  20.   //
  21.   // +optional
  22.   GatewayRefs []GatewayReference `json:"gatewayRefs,omitempty"`
  23. }

Added

To Route Specs:

  1.   // ParentRefs references the resources that can attach to this Route. The only
  2.   // kind of parent resource with "Core" support is Gateway. This API may be
  3.   // extended in the future to support additional kinds of parent resources such
  4.   // as one of the route kinds. It is invalid to reference an identical parent
  5.   // more than once. It is valid to reference multiple distinct sections within
  6.   // the same parent resource, such as 2 Listeners within a Gateway.
  7.   //
  8.   // It is possible to separately reference multiple distinct objects that may
  9.   // be collapsed by an implementation. For example, some implementations may
  10.   // choose to merge compatible Gateway Listeners together. If that is the case,
  11.   // the list of routes attached to those resources should also be merged.
  12.   //
  13.   // +optional
  14.   // +kubebuilder:validation:MaxItems=16
  15.   ParentRefs []ParentRef `json:"parentRefs,omitempty"`

And the struct that references:

  1. // ParentRef identifies an API object that should be considered a parent of this
  2. // resource. The only kind of parent resource with "Core" support is Gateway.
  3. // This API may be extended in the future to support additional kinds of parent
  4. // resources, such as HTTPRoute.
  5. type ParentRef struct {
  6.   // Group is the group of the referent.
  7.   //
  8.   // Support: Core
  9.   //
  10.   // +kubebuilder:validation:MinLength=1
  11.   // +kubebuilder:validation:MaxLength=253
  12.   // +kubebuilder:default=gateway.networking.k8s.io
  13.   Group string `json:"group"`
  15.   // Kind is kind of the referent.
  16.   //
  17.   // Support: Core (Gateway)
  18.   // Support: Extended (Other Resources)
  19.   //
  20.   // +kubebuilder:validation:MinLength=1
  21.   // +kubebuilder:validation:MaxLength=253
  22.   // +kubebuilder:default=Gateway
  23.   // +optional
  24.   Kind *string `json:"kind,omitempty"`
  26.   // Namespace is the namespace of the referent. When unspecified (empty
  27.   // string), this will either be:
  28.   //
  29.   // * local namespace of the target is a namespace scoped resource
  30.   // * no namespace (not applicable) if the target is cluster-scoped.
  31.   //
  32.   // Support: Extended
  33.   //
  34.   // +kubebuilder:validation:MinLength=1
  35.   // +kubebuilder:validation:MaxLength=253
  36.   // +optional
  37.   Namespace *string `json:"namespace,omitempty"`
  39.   // Scope represents if this refers to a cluster or namespace scoped resource.
  40.   // This may be set to "Cluster" or "Namespace".
  41.   //
  42.   // Support: Core (Namespace)
  43.   // Support: Extended (Cluster)
  44.   //
  45.   // +kubebuilder:validation:Enum=Cluster;Namespace
  46.   // +kubebuilder:default=Namespace
  47.   // +optional
  48.   Scope *string `json:"scope,omitempty"`
  50.   // Name is the name of the referent.
  51.   //
  52.   // Support: Core
  53.   //
  54.   // +kubebuilder:validation:MinLength=1
  55.   // +kubebuilder:validation:MaxLength=253
  56.   Name string `json:"name"`
  58.   // SectionName is the name of a section within the target resource. In the
  59.   // following resources, SectionName is interpreted as the following:
  60.   //
  61.   // * Gateway: Listener Name
  62.   //
  63.   // Implementations MAY choose to support attaching Routes to other resources.
  64.   // If that is the case, they MUST clearly document how SectionName is
  65.   // interpreted.
  66.   //
  67.   // When unspecified (empty string), this will reference the entire resource.
  68.   // For the purpose of status, an attachment is considered successful if at
  69.   // least one section in the parent resource accepts it. For example, Gateway
  70.   // listeners can restrict which Routes can bind to them by Route kind,
  71.   // namespace, or hostname. If 1 of 2 Gateway listeners accept attachment from
  72.   // the referencing Route, the Route MUST be considered successfully
  73.   // attached. If no Gateway listeners accept attachment from this Route, the
  74.   // Route MUST be considered detached from the Gateway.
  75.   //
  76.   // Support: Core
  77.   //
  78.   // +kubebuilder:validation:MinLength=1
  79.   // +kubebuilder:validation:MaxLength=253
  80.   // +optional
  81.   SectionName *string `json:"sectionName,omitempty"`
  82. }

Changed

To accommodate Routes with arbitrary types of parents, RouteGatewayStatus would be renamed to RouteParentStatus. Similarly, the GatewayRef inside that struct would be replaced with the ParentRef struct included above.

Advantages

  • Simplifies the API by providing a single way to attach Routes to Gateway.
  • Assigns clear responsibilities to Gateway and Route owners.
  • Consistent with pattern of direct references from Routes to all associated resources.
  • Enables attaching Routes to arbitrary parents, such as custom Gateways, other Routes (to be defined), or Meshes.
  • Prevents accidental exposure of Routes.
  • Easy to understand which Gateways/parents a Route is attached to.
  • Further simplifies path to Route inclusion.
  • Follows pattern of direct reference in one direction with a broader trust reference in the other direction.
  • Aligns with initial user feedback.

Disadvantages

  • Attaching a Route to a named listener with SectionName may be a bit confusing.
  • Does not utilize existing ReferenceGrant mechanism.
  • May be more difficult to understand which Routes are attached to a Gateway.
  • Adding/replacing a Gateway requires changes to Routes.

Potential Expansion

In the future, it may be useful to add a selector from Route -> Parent. Although this would enable greater flexibility, it also significantly increases complexity.

Alternatives

1. ReferenceGrant with Gateways selecting Routes

ReferenceGrant with Gateways selecting Routes

A compelling alternative to this proposal would involve retaining the Route selector in Gateway and replacing the trust concept in Routes with ReferenceGrant. To represent the same example as above, we’d use a Route selector on Gateway, a corresponding label on the HTTPRoute, and a ReferenceGrant that allowed it:

  1. kind: Gateway
  2. metadata:
  3.   name: xlb
  4.   namespace: infra
  5. spec:
  6.   listeners:
  7.   - name: foo
  8.     hostname: foo.com
  9.     port: 80
  10.     routes:
  11.       kind: HTTPRoute
  12.       selector:
  13.         gateway: xlb
  14.       namespaces:
  15.         from: Selector
  16.         selector:
  17.           kubernetes.io/metadata.name: foo
  18. ---
  19. kind: HTTPRoute
  20. metadata:
  21.   name: foo
  22.   namespace: foo
  23.   labels:
  24.     gateway: xlb
  25. spec:
  26.   rules:
  27.   - name: abc
  28.     matches:
  29.     - path: /bar
  30. ---
  31. kind: ReferenceGrant
  32. metadata:
  33.   name: infra-gateways
  34.   namespace: foo
  35. spec:
  36.   from:
  37.   - group: gateway.networking.k8s.io
  38.     kind: Gateway
  39.     namespace: infra
  40.   to:
  41.   - group: gateway.networking.k8s.io
  42.     kind: HTTPRoute

Advantages

  • Consistent use of ReferenceGrant throughout the API.
  • Provides a single way of binding Gateways to Routes.

Disadvantages

  • Even the simplest cross-namespace reference from Gateway -> Route would require a ReferenceGrant in each target namespace.
  • Existing demos and examples would become significantly more verbose.
  • Does not support attaching Routes to arbitrary parents.
  • Does not prevent accidental exposure of Routes.
  • Route owners have limited control in terms of which Gateways their Route is attached to.

2. ReferenceGrant with Routes referencing Gateways

ReferenceGrant with Routes referencing Gateways

The other way we could use ReferenceGrant would be with Routes referencing Gateways. Unfortunately the nested structure of Gateways makes this nearly impossible to do effectively. A core concept for Gateways is that each listener should be able to attach to an entirely different set of Routes. For example, a Gateway may want to delegate foo.com to the foo namespace and bar.com to the bar namespace. Unfortunately that would be very difficult to recreate with ReferenceGrant.

ReferenceGrant is fundamentally about trusting references from resource of kind Foo in to resources of kind Bar. Names and section names are intentionally excluded. If we added both of those concepts to ReferenceGrant, this would be possible, but quite complex and verbose. This is what the example from above would look like with this approach:

  1. kind: Gateway
  2. metadata:
  3.   name: lb
  4.   namespace: infra
  5. spec:
  6.   listeners:
  7.   - name: foo
  8.     hostname: foo.com
  9.     port: 80
  10. ---
  11. kind: ReferenceGrant
  12. metadata:
  13.   name: foo-lb
  14.   namespace: infra
  15. spec:
  16.   from:
  17.   - group: gateway.networking.k8s.io
  18.     kind: HTTPRoute
  19.     namespace: foo
  20.   to:
  21.   - group: gateway.networking.k8s.io
  22.     kind: Gateway
  23.     name: lb
  24.     sectionName: foo
  25. ---
  26. kind: HTTPRoute
  27. metadata:
  28.   name: foo
  29.   namespace: foo
  30. spec:
  31.   parentRefs:
  32.   - kind: Gateway
  33.     namespace: infra
  34.     name: lb
  35.     sectionName: foo
  36.   rules:
  37.   - name: abc
  38.     matches:
  39.     - path: /bar

Advantages

  • Consistent use of ReferenceGrant throughout the API.
  • Provides a single way of binding Gateways to Routes.
  • Supports attaching Routes to arbitrary parents.
  • Prevents accidental exposure of Routes.

Disadvantages

  • In most cases, each listener in a Gateway would require a unique ReferenceGrant resource.
  • Even the simplest cross-namespace reference from Route -> Gateway would require a ReferenceGrant in each target namespace. This could either rule out or significantly complicate self-service use-cases.
  • Existing demos and examples would become significantly more verbose.
  • ReferenceGrant would become more complex for all other use cases.

References

GEPs

Docs:

Issues: