Manual mode with short-term credentials for components

Manual mode with short-term credentials for components

During installation, you can configure the Cloud Credential Operator (CCO) to operate in manual mode and use the CCO utility (ccoctl) to implement short-term security credentials for individual components that are created and managed outside the OKD cluster.

This credentials strategy is supported for Amazon Web Services (AWS), Google Cloud Platform (GCP), and global Microsoft Azure only. The strategy must be configured during installation of a new OKD cluster. You cannot configure an existing cluster that uses a different credentials strategy to use this feature.

Cloud providers use different terms for their implementation of this authentication method.

Table 1. Short-term credentials provider terminology
Cloud provider	Provider nomenclature
Amazon Web Services (AWS)	AWS Security Token Service (STS)
Google Cloud Platform (GCP)	GCP Workload Identity
Global Microsoft Azure	Azure AD Workload Identity

AWS Security Token Service

In manual mode with STS, the individual OKD cluster components use the AWS Security Token Service (STS) to assign components IAM roles that provide short-term, limited-privilege security credentials. These credentials are associated with IAM roles that are specific to each component that makes AWS API calls.

Additional resources

Configuring an AWS cluster to use short-term credentials

AWS Security Token Service authentication process

The following diagram details the authentication flow between AWS and the OKD cluster when using AWS STS.

Figure 1. AWS Security Token Service authentication flow

AWS component secret formats

Using manual mode with the AWS Security Token Service (STS) changes the content of the AWS credentials that are provided to individual OKD components. Compare the following secret formats:

AWS secret format using long-term credentials

apiVersion: v1
kind: Secret
metadata:
  namespace: <target_namespace> (1)
  name: <target_secret_name> (2)
data:
  aws_access_key_id: <base64_encoded_access_key_id>
  aws_secret_access_key: <base64_encoded_secret_access_key>

1	The namespace for the component.
2	The name of the component secret.

AWS secret format using AWS STS

apiVersion: v1
kind: Secret
metadata:
  namespace: <target_namespace> (1)
  name: <target_secret_name> (2)
stringData:
  credentials: |-
    [default]
    sts_regional_endpoints = regional
    role_name: <operator_role_name> (3)
    web_identity_token_file: <path_to_token> (4)

1	The namespace for the component.
2	The name of the component secret.
3	The IAM role for the component.
4	The path to the service account token inside the pod. By convention, this is `/var/run/secrets/openshift/serviceaccount/token` for OKD components.

AWS component secret permissions requirements

OKD components require the following permissions. These values are in the CredentialsRequest custom resource (CR) for each component.

These permissions apply to all resources. Unless specified, there are no request conditions on these permissions.

Component	Custom resource	Required permissions for services
Cluster CAPI Operator	`openshift-cluster-api-aws`	EC2 `ec2:CreateTags` `ec2:DescribeAvailabilityZones` `ec2:DescribeDhcpOptions` `ec2:DescribeImages` `ec2:DescribeInstances` `ec2:DescribeInternetGateways` `ec2:DescribeSecurityGroups` `ec2:DescribeSubnets` `ec2:DescribeVpcs` `ec2:DescribeNetworkInterfaces` `ec2:DescribeNetworkInterfaceAttribute` `ec2:ModifyNetworkInterfaceAttribute` `ec2:RunInstances` `ec2:TerminateInstances` Elastic load balancing `elasticloadbalancing:DescribeLoadBalancers` `elasticloadbalancing:DescribeTargetGroups` `elasticloadbalancing:DescribeTargetHealth` `elasticloadbalancing:RegisterInstancesWithLoadBalancer` `elasticloadbalancing:RegisterTargets` `elasticloadbalancing:DeregisterTargets` Identity and Access Management (IAM) `iam:PassRole` `iam:CreateServiceLinkedRole` Key Management Service (KMS) `kms:Decrypt` `kms:Encrypt` `kms:GenerateDataKey` `kms:GenerateDataKeyWithoutPlainText` `kms:DescribeKey` `kms:RevokeGrant`^[1] `kms:CreateGrant` ^[1] `kms:ListGrants` ^[1]
Machine API Operator	`openshift-machine-api-aws`	EC2 `ec2:CreateTags` `ec2:DescribeAvailabilityZones` `ec2:DescribeDhcpOptions` `ec2:DescribeImages` `ec2:DescribeInstances` `ec2:DescribeInstanceTypes` `ec2:DescribeInternetGateways` `ec2:DescribeSecurityGroups` `ec2:DescribeRegions` `ec2:DescribeSubnets` `ec2:DescribeVpcs` `ec2:RunInstances` `ec2:TerminateInstances` Elastic load balancing `elasticloadbalancing:DescribeLoadBalancers` `elasticloadbalancing:DescribeTargetGroups` `elasticloadbalancing:DescribeTargetHealth` `elasticloadbalancing:RegisterInstancesWithLoadBalancer` `elasticloadbalancing:RegisterTargets` `elasticloadbalancing:DeregisterTargets` Identity and Access Management (IAM) `iam:PassRole` `iam:CreateServiceLinkedRole` Key Management Service (KMS) `kms:Decrypt` `kms:Encrypt` `kms:GenerateDataKey` `kms:GenerateDataKeyWithoutPlainText` `kms:DescribeKey` `kms:RevokeGrant`^[1] `kms:CreateGrant` ^[1] `kms:ListGrants` ^[1]
Cloud Credential Operator	`cloud-credential-operator-iam-ro`	Identity and Access Management (IAM) `iam:GetUser` `iam:GetUserPolicy` `iam:ListAccessKeys`
Cluster Image Registry Operator	`openshift-image-registry`	S3 `s3:CreateBucket` `s3:DeleteBucket` `s3:PutBucketTagging` `s3:GetBucketTagging` `s3:PutBucketPublicAccessBlock` `s3:GetBucketPublicAccessBlock` `s3:PutEncryptionConfiguration` `s3:GetEncryptionConfiguration` `s3:PutLifecycleConfiguration` `s3:GetLifecycleConfiguration` `s3:GetBucketLocation` `s3:ListBucket` `s3:GetObject` `s3:PutObject` `s3:DeleteObject` `s3:ListBucketMultipartUploads` `s3:AbortMultipartUpload` `s3:ListMultipartUploadParts`
Ingress Operator	`openshift-ingress`	Elastic load balancing `elasticloadbalancing:DescribeLoadBalancers` Route 53 `route53:ListHostedZones` `route53:ListTagsForResources` `route53:ChangeResourceRecordSets` Tag `tag:GetResources` Security Token Service (STS) `sts:AssumeRole`
Cluster Network Operator	`openshift-cloud-network-config-controller-aws`	EC2 `ec2:DescribeInstances` `ec2:DescribeInstanceStatus` `ec2:DescribeInstanceTypes` `ec2:UnassignPrivateIpAddresses` `ec2:AssignPrivateIpAddresses` `ec2:UnassignIpv6Addresses` `ec2:AssignIpv6Addresses` `ec2:DescribeSubnets` `ec2:DescribeNetworkInterfaces`
AWS Elastic Block Store CSI Driver Operator	`aws-ebs-csi-driver-operator`	EC2 `ec2:AttachVolume` `ec2:CreateSnapshot` `ec2:CreateTags` `ec2:CreateVolume` `ec2:DeleteSnapshot` `ec2:DeleteTags` `ec2:DeleteVolume` `ec2:DescribeInstances` `ec2:DescribeSnapshots` `ec2:DescribeTags` `ec2:DescribeVolumes` `ec2:DescribeVolumesModifications` `ec2:DetachVolume` `ec2:ModifyVolume` `ec2:DescribeAvailabilityZones` `ec2:EnableFastSnapshotRestores` Key Management Service (KMS) `kms:ReEncrypt*` `kms:Decrypt` `kms:Encrypt` `kms:GenerateDataKey` `kms:GenerateDataKeyWithoutPlainText` `kms:DescribeKey` `kms:RevokeGrant`^[1] `kms:CreateGrant` ^[1] `kms:ListGrants` ^[1]

Request condition: kms:GrantIsForAWSResource: true

OLM-managed Operator support for authentication with AWS STS

In addition to OKD cluster components, some Operators managed by the Operator Lifecycle Manager (OLM) on AWS clusters can use manual mode with STS. These Operators authenticate with limited-privilege, short-term credentials that are managed outside the cluster. To determine if an Operator supports authentication with AWS STS, see the Operator description in OperatorHub.

Additional resources

CCO-based workflow for OLM-managed Operators with AWS STS

GCP Workload Identity

In manual mode with GCP Workload Identity, the individual OKD cluster components use the GCP workload identity provider to allow components to impersonate GCP service accounts using short-term, limited-privilege credentials.

Additional resources

Configuring a GCP cluster to use short-term credentials

GCP Workload Identity authentication process

The following diagram details the authentication flow between GCP and the OKD cluster when using GCP Workload Identity.

Figure 2. GCP Workload Identity authentication flow

GCP component secret formats

Using manual mode with GCP Workload Identity changes the content of the GCP credentials that are provided to individual OKD components. Compare the following secret content:

GCP secret format

apiVersion: v1
kind: Secret
metadata:
  namespace: <target_namespace> (1)
  name: <target_secret_name> (2)
data:
  service_account.json: <service_account> (3)

1	The namespace for the component.
2	The name of the component secret.
3	The Base64 encoded service account.

Content of the Base64 encoded service_account.json file using long-term credentials

{
   "type": "service_account", (1)
   "project_id": "<project_id>",
   "private_key_id": "<private_key_id>",
   "private_key": "<private_key>", (2)
   "client_email": "<client_email_address>",
   "client_id": "<client_id>",
   "auth_uri": "https://accounts.google.com/o/oauth2/auth",
   "token_uri": "https://oauth2.googleapis.com/token",
   "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
   "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/<client_email_address>"
}

1	The credential type is `service_account`.
2	The private RSA key that is used to authenticate to GCP. This key must be kept secure and is not rotated.

Content of the Base64 encoded service_account.json file using GCP Workload Identity

{
   "type": "external_account", (1)
   "audience": "//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/test-pool/providers/test-provider", (2)
   "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
   "token_url": "https://sts.googleapis.com/v1/token",
   "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/<client_email_address>:generateAccessToken", (3)
   "credential_source": {
      "file": "<path_to_token>", (4)
      "format": {
         "type": "text"
      }
   }
}

1	The credential type is `external_account`.
2	The target audience is the GCP Workload Identity provider.
3	The resource URL of the service account that can be impersonated with these credentials.
4	The path to the service account token inside the pod. By convention, this is `/var/run/secrets/openshift/serviceaccount/token` for OKD components.

Azure AD Workload Identity

In manual mode with Azure AD Workload Identity, the individual OKD cluster components use the Azure AD workload identity provider to assign components short-term security credentials.

Additional resources

Configuring a global Microsoft Azure cluster to use short-term credentials

Azure AD Workload Identity authentication process

The following diagram details the authentication flow between Azure and the OKD cluster when using Azure AD Workload Identity.

Figure 3. Azure AD Workload Identity authentication flow

Azure component secret formats

Using manual mode with AD Workload Identity changes the content of the Azure credentials that are provided to individual OKD components. Compare the following secret formats:

Azure secret format using long-term credentials

apiVersion: v1
kind: Secret
metadata:
  namespace: <target_namespace> (1)
  name: <target_secret_name> (2)
data:
  azure_client_id: <client_id> (3)
  azure_client_secret: <client_secret> (4)
  azure_region: <region>
  azure_resource_prefix: <resource_group_prefix> (5)
  azure_resourcegroup: <resource_group_prefix>-rg (6)
  azure_subscription_id: <subscription_id>
  azure_tenant_id: <tenant_id>
type: Opaque

1	The namespace for the component.
2	The name of the component secret.
3	The client ID of the Azure AD identity that the component uses to authenticate.
4	The component secret that is used to authenticate with Azure AD for the `<client_id>` identity.
5	The resource group prefix.
6	The resource group. This value is formed by the `<resource_group_prefix>` and the suffix `-rg`.

Azure secret format using AD Workload Identity

apiVersion: v1
kind: Secret
metadata:
  namespace: <target_namespace> (1)
  name: <target_secret_name> (2)
data:
  azure_client_id: <client_id> (3)
  azure_federated_token_file: <path_to_token_file> (4)
  azure_region: <region>
  azure_subscription_id: <subscription_id>
  azure_tenant_id: <tenant_id>
type: Opaque

1	The namespace for the component.
2	The name of the component secret.
3	The client ID of the user-assigned managed identity that the component uses to authenticate.
4	The path to the mounted service account token file.

Azure component secret permissions requirements

OKD components require the following permissions. These values are in the CredentialsRequest custom resource (CR) for each component.

Component	Custom resource	Required permissions for services
Cloud Controller Manager Operator	`openshift-azure-cloud-controller-manager`	`Microsoft.Compute/virtualMachines/read` `Microsoft.Network/loadBalancers/read` `Microsoft.Network/loadBalancers/write` `Microsoft.Network/networkInterfaces/read` `Microsoft.Network/networkSecurityGroups/read` `Microsoft.Network/networkSecurityGroups/write` `Microsoft.Network/publicIPAddresses/join/action` `Microsoft.Network/publicIPAddresses/read` `Microsoft.Network/publicIPAddresses/write`
Cluster CAPI Operator	`openshift-cluster-api-azure`	role: `Contributor` ^[1]
Machine API Operator	`openshift-machine-api-azure`	`Microsoft.Compute/availabilitySets/delete` `Microsoft.Compute/availabilitySets/read` `Microsoft.Compute/availabilitySets/write` `Microsoft.Compute/diskEncryptionSets/read` `Microsoft.Compute/disks/delete` `Microsoft.Compute/galleries/images/versions/read` `Microsoft.Compute/skus/read` `Microsoft.Compute/virtualMachines/delete` `Microsoft.Compute/virtualMachines/extensions/delete` `Microsoft.Compute/virtualMachines/extensions/read` `Microsoft.Compute/virtualMachines/extensions/write` `Microsoft.Compute/virtualMachines/read` `Microsoft.Compute/virtualMachines/write` `Microsoft.ManagedIdentity/userAssignedIdentities/assign/action` `Microsoft.Network/applicationSecurityGroups/read` `Microsoft.Network/loadBalancers/backendAddressPools/join/action` `Microsoft.Network/loadBalancers/read` `Microsoft.Network/loadBalancers/write` `Microsoft.Network/networkInterfaces/delete` `Microsoft.Network/networkInterfaces/join/action` `Microsoft.Network/networkInterfaces/loadBalancers/read` `Microsoft.Network/networkInterfaces/read` `Microsoft.Network/networkInterfaces/write` `Microsoft.Network/networkSecurityGroups/read` `Microsoft.Network/networkSecurityGroups/write` `Microsoft.Network/publicIPAddresses/delete` `Microsoft.Network/publicIPAddresses/join/action` `Microsoft.Network/publicIPAddresses/read` `Microsoft.Network/publicIPAddresses/write` `Microsoft.Network/routeTables/read` `Microsoft.Network/virtualNetworks/delete` `Microsoft.Network/virtualNetworks/read` `Microsoft.Network/virtualNetworks/subnets/join/action` `Microsoft.Network/virtualNetworks/subnets/read` `Microsoft.Resources/subscriptions/resourceGroups/read`
Cluster Image Registry Operator	`openshift-image-registry-azure`	Data permissions `Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete` `Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write` `Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read` `Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action` `Microsoft.Storage/storageAccounts/blobServices/containers/blobs/move/action` General permissions `Microsoft.Storage/storageAccounts/blobServices/read` `Microsoft.Storage/storageAccounts/blobServices/containers/read` `Microsoft.Storage/storageAccounts/blobServices/containers/write` `Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action` `Microsoft.Storage/storageAccounts/read` `Microsoft.Storage/storageAccounts/write` `Microsoft.Storage/storageAccounts/delete` `Microsoft.Storage/storageAccounts/listKeys/action` `Microsoft.Resources/tags/write`
Ingress Operator	`openshift-ingress-azure`	`Microsoft.Network/dnsZones/A/delete` `Microsoft.Network/dnsZones/A/write` `Microsoft.Network/privateDnsZones/A/delete` `Microsoft.Network/privateDnsZones/A/write`
Cluster Network Operator	`openshift-cloud-network-config-controller-azure`	`Microsoft.Network/networkInterfaces/read` `Microsoft.Network/networkInterfaces/write` `Microsoft.Compute/virtualMachines/read` `Microsoft.Network/virtualNetworks/read` `Microsoft.Network/virtualNetworks/subnets/join/action` `Microsoft.Network/loadBalancers/backendAddressPools/join/action`
Azure File CSI Driver Operator	`azure-file-csi-driver-operator`	`Microsoft.Network/networkSecurityGroups/join/action` `Microsoft.Network/virtualNetworks/subnets/read` `Microsoft.Network/virtualNetworks/subnets/write` `Microsoft.Storage/storageAccounts/delete` `Microsoft.Storage/storageAccounts/fileServices/read` `Microsoft.Storage/storageAccounts/fileServices/shares/delete` `Microsoft.Storage/storageAccounts/fileServices/shares/read` `Microsoft.Storage/storageAccounts/fileServices/shares/write` `Microsoft.Storage/storageAccounts/listKeys/action` `Microsoft.Storage/storageAccounts/read` `Microsoft.Storage/storageAccounts/write`
Azure Disk CSI Driver Operator	`azure-disk-csi-driver-operator`	`Microsoft.Compute/disks/` `Microsoft.Compute/snapshots/` `Microsoft.Compute/virtualMachineScaleSets//read` `Microsoft.Compute/virtualMachineScaleSets/read` `Microsoft.Compute/virtualMachineScaleSets/virtualMachines/write` `Microsoft.Compute/virtualMachines/``/read` `Microsoft.Compute/virtualMachines/write` `Microsoft.Resources/subscriptions/resourceGroups/read`

This component requires a role rather than a set of permissions.