GEP-1364: Status and Conditions Update

• Issue: #1364
• Status: Standard

TLDR

The status, particularly the Conditions, across the whole Gateway API have very much grown organically, and so have many inconsistencies and odd behaviors. This GEP covers doing a review and consolidation to make Condition behavior consistent across the whole API.

Goals

• Update Conditions design to be consistent across Gateway API resources
• Provide a model and guidelines for Conditions for future new resources
• Specify changes to conformance required for Condition updates

Non-Goals

• Define the full set of Conditions that will ever be used with Gateway API

Introduction

Gateway API currently has a lot of issues related to status, especially that status is inconsistent (#1111), that names are hard to understand (#1110), and that Reasons aren’t explained properly (#1362).

As the API has grown, the way we talk about resources has changed a lot, and some of the status design hasn’t been updated since resources were created.

So, for example, we have GatewayClass with Accepted, Gateway with Scheduled, the Gateway Listeners with Detached (which you want to be false, unlike the previous two), and then Gateways and Gateway Listeners have Ready, but Route doesn’t (and which also you want to be true).

This document lays out large-scale changes to the way that we talk about resources, and the Conditions to match them. This means that there will be an unavoidable break in what constitutes a healthy or unhealthy resource, and there will be changes required for all implementations to be conformant with the release that includes these changes.

The constants that mark the deprecated types will be also marked as deprecated, and will no longer be tested as part of conformance. They’ll still be present, and will work, but they won’t be part of the spec any more. This should give implementations and users a release to transition to the new design (in UX terms). This grace period should be one release (so, the constants will be removed in v0.7.0.)

This level of change is not optimal, and the intent is to make this a one-off change that can be built upon for future resources - since there are definitely more resources on the way.

Background: Kubernetes API conventions and prior art on Conditions

Because this GEP is mainly concerned with updating the Conditions we are setting in Gateway API resources’ status, it’s worth reviewing some important points about Conditions. (This information is mainly taken from the Typical status properties section of the API conventions document.)

1. Conditions are a standard type used to represent arbitrary higher-level status from a controller.
2. They are a listMapType, a list that is enforced by the apiserver to have only one entry of each item, using the type field as a key. (So, this is effectively a map that looks like a list in YAML form).

3. Each has a number of fields, the most important of which for this discussion are type, status, reason, and observedGeneration.

   • type is a string value indicating the Condition type. Accepted, Scheduled, and Ready are current examples.
   • status indicates the state of the condition, and can be one of three values, true, false, or unknown. Unknown in particular is important, because it means that the controller is unable to determine the status for some reason. (Also notable is that “” is also valid, and must be treated as Unknown. Controllers must not set the value to “”, but consumers should accept it as meaning the same thing as Unknown.)
   • reason is a CamelCase string that is a brief description of the reason why the status is set the way it is.
   • observedGeneration is an optional field that sets what the metadata.generation field was when the controller last saw a resource. Note that this is optional in the struct, but is required for Gateway API conditions. This will be enforced in the conformance tests in the future.

4. Conditions should describe the current state of the resource at observation time, which means that they should be an adjective (like Ready), or a past-tense verb (like Accepted). This one in particular is documented pretty closely in the Typical status properties section of the guidelines.

5. Conditions should be applied to a resource the first time the controller sees the resource. This seems to imply that all conditions should be present on every resource owned by a controller, but the rest of the conventions don’t make this clear, and it is often not complied with.

6. It’s helpful to have a top-level condition which summarizes more detailed conditions. The guidelines suggest using either Ready for long-running processes, or Succeeded for bounded execution.

From these guidelines, we can see that Conditions can be either positive polarity (healthy resources have them as status: true) or negative polarity (healthy resources have them as status: false). Ready is an example of a positive polarity condition, and conditions like Conflicted from Listener or NetworkUnavailable, MemoryPressure, or DiskPressure from the Node resource are examples of negative-polarity conditions.

There is also some extra context that’s not in the API conventions doc:

SIG-API Machinery has been reluctant to add fields that would aid in machine-parsing of Conditions, especially fields that would indicate the polarity, because they are intended more for human consumption than machine consumption. Probably the best example of this was in the PR #4521.

This means that there’s no guidance from upstream about condition polarity. We’ll discuss this more when we talk about new conditions.

The guidance about Conditions being added as soon as a controller sees a resource is a bit unclear - as written in the conventions, it seems to imply that all relevant conditions should always be added, even if their status has to be set to unknown. Gateway API resources do not currently require this, and the practice seems to be uncommon.

Proposed changes

Proposed changes summary

• All the current Conditions that indicate that the resource is okay and ready for processing will be replaced with Accepted.
• In general, resources should be considered Accepted if their config is valid enough to generate some config in the underlying data plane. Examples are provided below.
• There will be a limited set of positive polarity summary conditions, and a number of other specific negative-polarity error conditions.
• All relevant positive-polarity summary Conditions for a resource must be added when it’s observed. For example, HTTPRoutes must always have Accepted and ResolvedRefs, regardless of their state.
• Negative polarity error conditions must only be added when the error is True.
• The Ready condition will be moved to Extended conformance, and we’ll re-evaluate if it’s used by any implementations after some time has passed. If not, it may be removed.
• To capture the behavior that Ready currently captures, Programmed will be introduced. This means that the implementation has seen the config, has everything it needs, parsed it, and sent configuration off to the data plane. The configuration should be available “soon”. We’ll leave “soon” undefined for now.
• Resolving a comment that came up, documentation will be added to clarify that it’s okay to add your own Conditions, and that implementations should namespace their custom Conditions with a domain prefix (so implementation.io/CustomType rather than just CustomType), or run the risk of using a word that’s reserved later.
• It’s recommended that implementations publish both new and old conditions to provide a smoother transition, but conformance tests will only require the new conditions.

The exact list of changes is detailed below. The next few sections detail the reasons for these large-scale changes.

Conceptual and language changes

Gateway API resources are, conceptually, all about breaking up the configuration for a data plane into separate resources that are expressive and extensible, while being split up along role-oriented boundaries.

So, when we talk about Gateway API, it’s always about a system of related resources.

We already acknowledge this when we talk about Routes “attaching” to Gateways, or Gateways referencing Services, or Gateways requiring a GatewayClass in their spec.

However, this GEP is proposing that we move all our discussion into using “accepted” to indicate that a resource has attached correctly enough to be accepted for processing.

So resources are Accepted for processing when their attachment succeeds enough to generate some configuration. This allows us to make calls about when partially valid objects should be accepted and when they shouldn’t.

Of course, because we’re using all of this configuration to describe some sort of data path from “outside”/lacking cluster context to “inside”/enriched with cluster context, we also need a way to describe when that data path is configured and working.

We already have a word in the Kubernetes API, but it comes with some expectations that implementations are not currently able to meet. That word is Ready, but it implies that the data path is Ready when you read the status, rather than that it will be ready soon (which is what most implementations can guarantee currently.)

So we have an unresolved question as to what to do with the Ready condition. This is addressed further below.

Condition polarity

In terms of the polarity of conditions, we have three options, of which only two are really viable: * All conditions must be negative polarity * All conditions must be positive polarity * Some conditions can be positive polarity, but most should be negative.

The fact that the user experience of Ready or conditions like Accepted being true in the healthy case is much better rules out the first option, so we are left to decide between enforcing that all conditions are positive, or that we have a mix.

Having an arbitrary mix will make doing machine-based extraction of information much harder, so here I’m going to talk about the distinction between having all conditions positive or some, summary conditions positive, and the rest negative.

All Conditions Positive

In this case, all Condition types are written in such a way that they’re positive polarity, and are true in the healthy case.

As already discussed, Ready, and Accepted are current examples, but another one that’s a little more important here is ResolvedRefs which is set to true when all references to other resources have been successfully resolved. This is not a blocking Condition that affects the Ready condition, since having some references valid is enough to produce some configuration in the underlying data plane.

So, All Conditions Positive pros:

• We’re close already. Most conditions in the API are currently positive polarity.
• Easier to understand - there are no double negatives. “Good: true” is less cognitive overhead than “NotGood: false”.

Cons:

• Reduces flexibility - it can surprisingly difficult to avoid double negatives for conditions that describe error states, as in general programmers are more used to reporting “something went wrong” than they are “everything’s okay”.

Not sure if pro or con:

• Leans the design towards favoring conditions always being present, since you can’t be sure if everything is good unless you see AllGood: true. The absence of a positive-polarity condition implies that the condition could be false. This puts this option more in line with the API guidelines on this point.

Some Conditions Positive

In this case, only a limited set of summary conditions are positive, and the rest are negative.

Pros:

• Error states can be described with Error: true instead of NoError: false.
• Negative polarity error conditions are more friendly to not being present (since absence of Error: true implies everything’s okay).

Cons:

• Any code handling conditions will need a list of the positive ones, and will need to assume that any others are negative.

Decision

Gateway API conditions will be positive for conditions that describe the happy state of the object, which is currently Accepted and ResolvedRefs, and will also include the new Programmed condition, and the newly-Extended condition Ready. A separate set of negative-polarity Error conditions will be set on an object when they are true.

Should conditions always be added?

Not all of them.

Positive polarity Conditions that describe the desirable state of the object must always be set. These are currently Accepted, ResolvedRefs, and Programmed. Implementations that use Ready must also add it before programming the Route.

Partial validity and Conditions

One of the trickiest parts of Gateway API objects is that it’s very possible to end up with an object that has some parts with valid configuration and some that don’t. We refer to this as partial validity, and communicating this via status conditions is difficult.

The intent with the Accepted condition is that it serves as an indicator that something is working, that some traffic from what the config specifies will be routed as configured.

At this time, we haven’t added a “no errors at all present” Condition, choosing to have a “some config is working” condition, with specific errors to aid in finding the exact problem with the objects. We could conceivably add this later if users find Accepted insufficient, but we’re erring on the side of having less positive Conditions for now.

New and Updated Conditions

Accepted

This GEP proposes replacing all conditions that indicate syntactic and semantic validity with one, Accepted condition type.

That is, the proposal is to replace:

• Scheduled on Gateway
• Detached on Listener

with Accepted in all these locations.

GatewayClass and Route will maintain the Accepted condition.

All of these conditions share the following meanings:

• The resource has been accepted for processing by the controller
• The resource is syntactically and semantically valid, and internally consistent
• The resource fits into a larger system of Gateway API resources, and there is is no missing information, including but not limited to:
• Any mandatory references resolve to existing resources (examples here are the Gateway’s gatewayClass field, or the parentRefs field in Route resources)
• Any specified TLS secrets exist
• The resource is supported by the controller by ensuring things like:
• Any Kinds being referred to by the resource are supported
• Features being used by the resource are supported

All of these rules can be summarized into:

• The resource is valid enough to produce some configuration in the underlying data plane.

For Gateway, Accepted also subsumes the functions of Scheduled: Accepted set to true means that sufficient capacity exists on underlying infrastructure for the Gateway to be provisioned. If that capacity does not exist, then the Gateway cannot be reconciled successfully, and so fails to attach to the owning GatewayClass, and cannot be accepted.

Note that some classes of inter-resource reference failure do not cause a resource to become unattached and stop being accepted (that is, to have the Accepted condition set to status: false).

• Non-existent Service backends - if the backend does not exist on a HTTPRoute that is otherwise okay, then the data plane must generate 500s for traffic that matches that HTTPRoute. In this case, the Accepted Condition must be true, and the ResolvedRefs Condition must be false, with reasons and messages indicating that the backend services do not exist.
• HTTPRoutes with all backends in other namespaces, but not permitted by ReferenceGrants. In this case, the “non-existent service backends” rules apply, and 500s must be generated. In this case, again, the Accepted condition is true, and the ResolvedRefs Condition is false, with reasons and messages indicating that the backend services are not reachable.

For ReferenceGrant or not-designed-yet Policy resources, Accepted means that:

• the resource has a correctly-defined set of resources that it applies to
• the resource has a syntactically and semantically valid spec

Note that having a correctly-defined set of resources that is empty does not make these resources unattached, as long as it’s possible to create some config in the underlying data plane. By “empty” here we mean that there are no backends, not that the config is incomplete or missing references. So you can have a GatewayClass, Gateway, HTTPRoute and Service all present and referred to correctly when there are no endpoints in the Service, and the resource will not stop being accepted, because HTTPRoute contains rules about what to program in the data plane if there are no endpoints (that is, it should return 500 for any matching request).

Note that for other Route types that don’t have a clear mechanism like HTTP does for indicating a server failure (like the HTTP code 500 does), not having existing backends may not produce any configuration in the data plane, and so may cause the resource to fail to attach. (An example here could be a TCP Route with no backends, we need to decide if that means that a port should be opened that actively closes connections, or if no port should be opened.)

Examples of Conditions:

• HTTPRoute with one match with one backend that is valid. Accepted is true, ResolvedRefs is true.
• HTTPRoute with one match with one backend that is a non-existent Service backend. The Accepted Condition is true, the ResolvedRefs condition is false, with a reason of BackendNotFound. Accepted is true in this case because the data path must respond to requests that would be sent to that backend with a 500 response.
• HTTPRoute with one match with two backends, one of which is a non-existent Service backend. The Accepted Condition is true, the ResolvedRefs condition is false. Accepted is true in this case because the data path must respond to a percentage of the requests matching the rule corresponding to the weighting of the non-existent backend (which would be fifty percent unless weights are applied).
• HTTPRoute with one match with one backend that is in a different namespace, and does not have a ReferenceGrant permitting that access. The Accepted condition is true, and the ResolvedRefs Condition is false, with a reason of RefNotPermitted. As before, Accepted is true because in this case, the data path must be programmed with 500s for the match.
• TCPRoute with one match with a backend that is a non-existent Service. Accepted is false, and ResolvedRefs is false. Accepted is false in this case because there is not enough information to program any rules to handle the traffic in the underlying data plane - TCP doesn’t have a way to say “this is a valid destination that has something wrong with it”.
• HTTPRoute with one Custom supported filter added that is not supported by the implementation. Our spec is currently unclear on what happens in this case, but custom HTTP Filters require the use of the ExtensionRef filter type, and the setting of the ExtensionRef field to the name, group, version, and kind of a custom resource that describes the filter. If that custom resource is not supported, it seems reasonable to say that this should be a reference failure, and be treated like other reference failures (Accepted will be set to true, ResolvedRefs to false with a InvalidKind Reason, and traffic that would have matched the filter should receive a 500 error.)
• A HTTPRoute with one rule that specifies a HTTPRequestRedirect filter and a HTTPURLRewrite filter. Accepted must be false, because there’s only one rule, and this configuration for the rule is invalid (see reference) The error condition in this case is undefined currently - we should define it, thanks @sunjayBhatia.
• A HTTPRoute with two rules, one valid and one which specifies a HTTPRequestRedirect filter _and a HTTPURLRewrite filter. Accepted is true, because the valid rule can produce some config in the data plane. We’ll need to raise the more specific error condition for an incompatible filter combination as well to make the partial validity clear.

Ready

Currently, the Ready condition text for Gateway says:

    // This condition is true when the Gateway is expected to be able
    // to serve traffic. Note that this does not indicate that the
    // Gateway configuration is current or even complete (e.g. the
    // controller may still not have reconciled the latest version,
    // or some parts of the configuration could be missing).

This is pretty unclear - how can the Gateway serve traffic if config is missing? In the past, we’ve been asked to have a Condition that only flips to true when all required configuration is present.

For many implementations (certainly for Envoy-based ones), getting this information correctly and avoiding races on applying it is surprisingly difficult.

For this reason, this GEP proposes that we exclude the Ready condition from Core conformance, and make it a feature that implementations may opt in to - making it an Extended condition.

It will have the following behavior:

• Ready is an optional Condition that has Extended support, with conformance tests to verify the behavior.
• When it’s set, the condition indicates that traffic is ready to flow through the data plane immediately, not at some eventual point in the future.

We’ll need to add conformance testing for this.

Programmed

The Programmed condition is being added to replicate the functionality that the Ready condition currently indicates, namely that all the resources in the set are valid enough to produce some data plane configuration, and that configuration has been sent to the data plane, and should be ready soon.

It is a positive-polarity summary condition, and so should always be present on the resource. It should be set to Unknown if the implementation performs updates to the status before it has all the information it needs to be able to determine if the condition is true.

Alternatives

(Most alternatives have been discussed inline. Please comment here if this section needs updating.)

References