Knative Serving API Specification

Authors: argent@google.com, dangerd@google.com

Abstract

The Knative serving platform provides common abstractions for managing request-driven, short-lived, stateless compute resources in the style of common FaaS and PaaS offerings. This document describes the structure, lifecycle and management of Knative resources in the context of the Kubernetes Resource Model. An understanding of the Kubernetes API interface and the capabilities of Kubernetes Custom Resources is assumed. The Knative Serving API surface aims to support the following additional constraints beyond the general Kubernetes model:

• Client-side orchestration should not be required for common operations such as deployment, rollback, or simple imperative updates (“change environment variable X”).
• Both declarative (aka GitOps) and imperative (command-driven) management of resources should be usable, though not on the same resources at the same time. (I.e. one application may use a checked-in configuration which is automatically pushed, while a different application may be pushed and updated by hand with the authoritative state living on the server.)
• Developers can effectively use the Knative Serving stack without needing to engage beyond the core Knative resource model. Conversely, platform operators may restrict developers (e.g. via RBAC) to only be able to operate on the core Knative Serving resources, and may provide additional abstractions to manage platform-specific settings.
• The Knative Serving API may be deployed in a single-tenant or multi-tenant environment; it is assumed that developers may not have access to multiple namespaces or any cluster-level resources.

This document does not define the runtime contract (see this document) nor prescribe specific implementations of supporting services such as access control, observability, or resource management.

This document makes reference in a few places to different profiles for Knative Serving installations. A profile in this context is a set of operations, resources, and fields that are accessible to a developer interacting with a Knative installation. Currently, only a single (minimal) profile for Knative Serving is defined, but additional profiles may be defined in the future to standardize advanced functionality. A minimal profile is one that implements all of the “MUST”, “MUST NOT”, and “REQUIRED” conditions of this document.

Background

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED”, “MAY”, and “OPTIONAL” are to be interpreted as described in RFC 2119.

There is no formal specification of the Kubernetes API and Resource Model. This document assumes Kubernetes 1.13 behavior; this behavior will typically be supported by many future Kubernetes versions. Additionally, this document may reference specific core Kubernetes resources; these references may be illustrative (i.e. an implementation on Kubernetes) or descriptive (i.e. this Kubernetes resource MUST be exposed). References to these core Kubernetes resources will be annotated as either illustrative or descriptive.

This document considers two users of a given Knative Serving environment, and is particularly concerned with the expectations of developers (and language and tooling developers, by extension) deploying applications to the environment.

• Developers write code which is packaged into a container which is run on the Knative Serving cluster.
  • Language and tooling developers typically write tools used by developers to package code into containers. As such, they are concerned that tooling which wraps developer code to produce resources which match the Knative API contract.
• Operators (also known as platform providers) provision the compute resources and manage the software configuration of Knative Serving and the underlying abstractions (for example: Linux, Kubernetes, Istio, etc).

RBAC Profile

In order to validate the controls described in Resource Overview, the following Kubernetes RBAC profile may be applied in a Kubernetes cluster. This Kubernetes RBAC is an illustrative example of the minimal profile rather than a requirement. This Role should be sufficient to develop, deploy, and manage a set of serving applications within a single namespace. Knative Conformance tests against “MUST”, “MUST NOT”, and “REQUIRED” conditions are expected to pass when using this profile:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: knative-developer
rules:
  - apiGroups: ["serving.knative.dev"]
    resources: ["services"]
    verbs: ["get", "list", "create", "update", "delete"]
  - apiGroups: ["serving.knative.dev"]
    resources: ["configurations", "routes", "revisions"]
    verbs: ["get", "list"]

Resource Overview

The Knative Serving API provides a set of Kubernetes Custom Resources to manage stateless request-triggered (i.e. on-demand) autoscaled containers. Knative Serving assumes the use of HTTP (including HTTP/2 and layered protocols such as gRPC) as a request transport. In addition to low-level scaling and routing objects, Knative Serving provides a high-level Service object to reduce the cognitive overhead for application developers – the Service object should provide sufficient controls to cover most of application deployment scenarios (by frequency).

Service

The Knative Service represents an instantiation of a single serverless container environment (e.g. a microservice) over time. As such, a Service includes both a network address by which the Service may be reached as well as the application code and configuration needed to run the Service. The following table details which operations must be made available to a developer accessing a Knative Service using a minimal profile:

^ Kubernetes only allows JSON merge patch for CRD types. It is recommended that if allowed, at least JSON Merge patch be made available. JSON Merge Patch Spec (RFC 7386)

Revision

The Knative Revision represents a stateless, autoscaling snapshot-in-time of application code and configuration. Revisions enable progressive rollout and rollback of application changes by changing the HTTP routing between Service names and Revision instances. As such, Revisions are generally immutable, except where they may reference mutable core Kubernetes resources such as ConfigMaps and Secrets. Revisions can also be mutated by changes in Revision defaults. Changes to defaults that mutate Revisions are generally syntactic and not semantic.

Developers MUST NOT be able to create Revisions or update Revision spec directly; Revisions MUST be created in response to updates to a Configuration spec. It is RECOMMENDED that developers are able to force the deletion of Revisions to both handle the possibility of leaked resources as well as for removal of known-bad Revisions to avoid future errors in managing the service.

The following table details which operations must be made available to a developer accessing a Knative Revision using a minimal profile:

^ Kubernetes only allows JSON merge patch for CRD types. It is recommended that if allowed, at least JSON Merge patch be made available. JSON Merge Patch Spec (RFC 7386)

Route

The Knative Route represents the current HTTP request routing state against a set of Revisions. To enable progressive rollout of serverless applications, a Route supports percentage-based request distribution across multiple application code and configuration states (Revisions).

Routes which are owned (controlled) by a Service SHOULD NOT be updated by developers; any changes will be reset by the Service controller.

The table below details which operations must be made available to a developer accessing a Knative Route using a minimal profile. For any non-minimal profile, the POST, PUT, or DELETE operations MUST be enabled as a group. This ensures that the developer has the ability to control the complete lifecycle of the object from create through deletion.

^ Kubernetes only allows JSON merge patch for CRD types. It is recommended that if allowed, at least JSON Merge patch be made available. JSON Merge Patch Spec (RFC 7386)

Configuration

The Knative Configuration represents the desired future state (after deployments complete) of a single autoscaled container application and configuration. The Configuration provides a template for creating Revisions as the desired state of the application changes.

Configurations which are owned (controlled) by a Service SHOULD NOT be updated by developers; any changes will be reset by the Service controller. These changes MAY still generate side effects such as the creation of additional Revisions.

The table below details which operations must be made available to a developer accessing a Knative Configuration using a minimal profile. For any advanced profile, the POST, PUT, or DELETE operations MUST be enabled as a group. This ensures that the developer has the ability to control the complete lifecycle of the object from create through deletion.

^ Kubernetes only allows JSON merge patch for CRD types. It is recommended that if allowed, at least JSON Merge patch be made available. JSON Merge Patch Spec (RFC 7386)

Error Signalling

The Knative API uses the Kubernetes Conditions convention to communicate errors and problems to the user. Each user-visible resource described in Resource Overview MUST have a conditions field in status, which must be a list of Condition objects of the following form (note that the actual API object types may be named FooCondition to allow better code generation and disambiguation between similar fields in the same apiGroup):

Additionally, the resource’s status.conditions field MUST be managed as follows to enable clients (particularly user interfaces) to present useful diagnostic and error message to the user. In the following section, conditions are referred to by their type (aka the string value of the type field on the Condition).

1. Each resource MUST have either a Ready condition (for ongoing systems) or Succeeded condition (for resources that run to completion) with severity="", which MUST use the True, False, and Unknown status values as follows:

   1. False MUST indicate a failure condition.
   2. Unknown SHOULD indicate that reconciliation is not yet complete and success or failure is not yet determined.
   3. True SHOULD indicate that the application is fully reconciled and operating correctly.

   Unknown and True are specified as SHOULD rather than MUST requirements because there may be errors which prevent serving which cannot be determined by the API stack (e.g. DNS record configuration in certain environments). Implementations are expected to treat these as “MUST” for factors within the control of the implementation.

2. For non-Ready conditions, any conditions with severity="" (aka “Error conditions”) must be aggregated into the “Ready” condition as follows:

   1. If the condition is False, Ready MUST be False.
   2. If the condition is Unknown, Ready MUST be False or Unknown.
   3. If the condition is True, Ready may be any of True, False, or Unknown.

   Implementations MAY choose to report that Ready is False or Unknown even if all Error conditions report a status of True (i.e. there may be additional hidden implementation conditions which feed into the Ready condition which are not reported.)

3. Non-Ready conditions with non-error severity MAY be surfaced by the implementation. Examples of Warning or Info conditions could include: missing health check definitions, scale-to-zero status, or non-fatal capacity limits.

Conditions type names should be chosen to describe positive conditions where True means that the condition has been satisfied. Some conditions may be transient (for example, ResourcesAllocated might change between True and False as an application scales to and from zero). It is RECOMMENDED that transient conditions be indicated with a severity="Info".

Resource Lifecycle

Revisions are created by updates to the spec of Configuration resources, which own the Revision lifecycle. Similarly, Service resources create and own (control) Routes and Configurations. In some profiles, Route and Configuration may be directly created without creating an owning Service; this is considered an advanced environment, as it exposes more concepts and higher-cardinality relationships to the developer.

This section describes the rules by which the different resources interact. The general Kubernetes model is that of continuous reconciliation across eventually-consistent resources.

Ownership

In several of the following sections, resources are said to “own” another resource. Ownership indicates that the owned resource is being managed by the owning resource and MUST be recorded using an OwnerReference in metadata.ownerReferences on the owned resource with controller set to True and uid set to the UID of the owning resource.

If the owning resource determines an ownership conflict and does not hold an OwnerReference, the owning resource SHOULD NOT modify the resource, and SHOULD signal the conflict with an error Condition.

Deleting an owning resource MUST trigger deletion of all owned resources; this deletion MAY be immediate or eventually consistent. This process can be implemented using Kubernetes cascading deletion.

Service

When a Service is created, it MUST create and own a Configuration and Route with the same name as the Service. Updates to spec, metadata.labels, and metadata.annotations of the Service MUST be copied to the appropriate Configuration or Route, as follows:

• metadata changes MUST be copied to both Configuration and Route.
• In addition, the serving.knative.dev/service label on the Route and Configuration MUST be set to the name of the Service.
• Additional labels and annotations on the Configuration and Route not specified above MUST be removed.
• See the documentation of spec in the detailed resource fields section for the mapping of specific spec fields to the corresponding fields in Configuration and Route.

Similarly, the Service MUST update its status fields based on the corresponding status of its owned Route and Configuration. The Service MUST include conditions of ConfigurationsReady and RoutesReady in addition to the generic Ready condition; other conditions MAY also be present.

Configuration

When a Configuration is created or its spec updated, the following steps MUST be taken to create a new Revision (on creation, the previous state for all fields is an empty value):

• If the spec.template field has changed from the previous state, a new owned Revision MUST be created. If the Revision name is not provided by the user through the spec.template.metadata.name field, it MUST be system generated. The system generated name should be treated opaquely as no semantics are defined. The values of spec.template.metadata and spec.template.spec MUST be copied to the newly created Revision. If the Revision cannot be created, the Configuration MUST signal that the Configuration is not Ready.
• metadata.labels and metadata.annotations from the Configuration MUST NOT be copied to the newly-created Revision.

Configuration MUST track the status of owned Revisions in order of creation, and report the name of the most recently created Revision in status.latestCreatedRevisionName and the name of the most recently created Revision where the Ready Condition is True in status.latestReadyRevisionName. These fields MAY be used by client software and Route objects to determine the status of the Configuration and the best target Revision for requests.

Revision

A Revision is automatically scaled by the underlying infrastructure; a Revision with no referencing Routes MAY be automatically scaled to zero instances and backing resources collected. Additionally, Revisions which are older than the oldest live Revision (referenced by at least one Route or latest for the Configuration) MAY be automatically deleted by the system.

Route

Route does not directly own any Knative resources, but may refer to multiple Revisions which will receive incoming requests either directly or through a reference to a Configuration’s status.latestReadyRevisionName field. In these cases, the Route MAY use a serving.knative.dev/route label to indicate that a Configuration or Revision is currently referenced by the Route. This label or other mechanisms MAY be used to prevent user deletion of Revisions referenced by a Route.

Request Routing

Knative uses fractional host-based HTTP routing to deliver requests to autoscaled instances. Knative Routes specify request routing from HTTP virtual hosts (Host-header based routing) to fractional assignments to Revisions. The following semantics apply to Knative request routing:

• Each HTTP/1.1 request MUST be treated as a separate “request” using the following semantics.
• If supported by the Knative installation, each HTTP/2 stream MUST be treated as a separate “request” using the following semantics. Note that this maps naturally for HTTP requests over HTTP/2, but is also well-defined for other applications like gRPC.
• DNS Hostnames allocated to Routes MUST be unique.
• DNS Hostnames allocated to Routes SHOULD resolve. Wildcard DNS records associated with a domain assigned to the namespace are the RECOMMENDED implementation.
• Requests to a specific Host MUST be dispatched to Revision instances in accordance with the weights specified in the Route, even if the number of container instances per Revision does not match the weights specified in the Route.
• When a selected Revision does not have available instances, the routing infrastructure MUST hold (delay) requests until an available instance is ready, scheduling a new container instance if needed. This is sometimes referred to as a “cold start”.
• Requests which cause a new container instance to be created (“cold start”) SHOULD be sent to the initially selected Revision, rather than to a different Revision.
• Multiple simultaneous or subsequent requests from a single client (even over the same TCP connection) MAY be dispatched to different instances or different Revisions (traffic routing MUST NOT be “sticky” in a way which violates the weight distributions). Developers SHOULD NOT assume that subsequent requests from the same client will reach the same application instance.

Detailed Resources – v1beta1

The following schema defines a set of REQUIRED or RECOMMENDED resource fields on the Knative resource types. Whether a field is REQUIRED or RECOMMENDED is denoted in the “Schema Requirement” column. Additional spec and status fields MAY be provided by particular implementations, however it is expected that most extension will be accomplished via the metadata.labels and metadata.annotations fields, as Knative implementations MAY validate supplied resources against these fields and refuse resources which specify unknown fields. Knative implementations MUST NOT require spec fields outside this implementation; to do so would break interoperability between such implementations and implementations which implement validation of field names.

Service

Metadata:

Standard Kubernetes meta.v1/ObjectMeta resource.

Service labels and annotations MUST be copied to the labels and annotations on the owned Configuration and Route. Additionally, the owned Configuration and Route MUST have the serving.knative.dev/service label set to the name of the Service.

Spec:

Status:

Configuration

Metadata:

Standard Kubernetes meta.v1/ObjectMeta resource.

Configuration labels and annotations MUST NOT be copied to the labels and annotations on newly-created Revisions. Configuration metadata MUST NOT be continuously copied to existing Revisions, which should remain immutable after creation. Additionally, the newly-created Revision MUST have the serving.knative.dev/configuration label set to the name of the Configuration. The Revision MUST also have the serving.knative.dev/configurationGeneration label set to the Configuration’s metadata.generation from which this Revision was created.

Spec:

Status:

Route

Metadata:

Standard Kubernetes meta.v1/ObjectMeta resource.

Spec:

Status:

Revision

Metadata:

Standard Kubernetes meta.v1/ObjectMeta resource.

Spec:

Status:

Detailed Resource Types - v1beta1

Although container, volumes, and types that they reference are based upon core Kubernetes objects, there are additional limitations applied to ensure that created containers can statelessly autoscale. The set of fields that have been determined to be compatible with statelessly scaling are detailed below. Restrictions to the values of the field are noted in the Description column.

ContainerStatuses

TrafficTarget

RevisionTemplateSpec

Container

EnvVar

EnvVarSource

ConfigMapKeySelector

SecretKeySelector

Probe

EnvFromSource

ExecAction

HTTPGetAction

TCPSocketAction

HTTPHeader

ContainerPort

ConfigMapEnvSource

SecretEnvSource

ResourceRequirements

SecurityContext

VolumeMount

Volume

ConfigMapVolumeSource

SecretVolumeSource

ProjectedVolumeSource

KeyToPath

VolumeProjection

ConfigMapProjection

SecretProjection

Addressable (Interface)

LocalObjectReference