Using manual mode with STS

Manual mode with STS is supported for Amazon Web Services (AWS).

In manual mode with STS, the individual OKD cluster components use AWS Secure Token Service (STS) to assign components IAM roles that provide short-term, limited-privilege security credentials. These credentials are associated with IAM roles that are specific to each component that makes AWS API calls.

Requests for new and refreshed credentials are automated by using an appropriately configured AWS IAM OpenID Connect (OIDC) identity provider, combined with AWS IAM roles. OKD signs service account tokens that are trusted by AWS IAM, and can be projected into a pod and used for authentication. Tokens are refreshed after one hour.

Detailed authentication flow between AWS and the cluster when using AWS STS

Figure 1. STS authentication flow

Using manual mode with STS changes the content of the AWS credentials that are provided to individual OKD components.

AWS secret format using long-lived credentials

  1. apiVersion: v1
  2. kind: Secret
  3. metadata:
  4.   namespace: <target-namespace> (1)
  5.   name: <target-secret-name> (2)
  6. data:
  7.   aws_access_key_id: <base64-encoded-access-key-id>
  8.   aws_secret_access_key: <base64-encoded-secret-access-key>
1The namespace for the component.
2The name of the component secret.

AWS secret format with STS

  1. apiVersion: v1
  2. kind: Secret
  3. metadata:
  4.   namespace: <target-namespace> (1)
  5.   name: <target-secret-name> (2)
  6. stringData:
  7.   credentials: |-
  8.     [default]
  9.     role_name: <operator-role-name> (3)
  10.     web_identity_token_file: <path-to-token> (4)
1The namespace for the component.
2The name of the component secret.
3The IAM role for the component.
4The path to the service account token inside the pod. By convention, this is /var/run/secrets/openshift/serviceaccount/token for OKD components.

Installing an OKD cluster configured for manual mode with STS

To install a cluster that is configured to use the Cloud Credential Operator (CCO) in manual mode with STS:

  1. Configure the Cloud Credential Operator utility.

  2. Create the required AWS resources individually, or with a single command.

  3. Run the OKD installer.

  4. Verify that the cluster is using short-lived credentials.

Because the cluster is operating in manual mode when using STS, it is not able to create new credentials for components with the permissions that they require. When upgrading to a different minor version of OKD, there are often new AWS permission requirements. Before upgrading a cluster that is using STS, the cluster administrator must manually ensure that the AWS permissions are sufficient for existing components and available to any new components.

Configuring the Cloud Credential Operator utility

To create and manage cloud credentials from outside of the cluster when the Cloud Credential Operator (CCO) is operating in manual mode with STS, extract and prepare the CCO utility (ccoctl) binary.

The ccoctl is a Linux binary that must run in a Linux environment.

Prerequisites

  • Obtain the OKD release image.

Procedure

  1. Get the CCO container image from the OKD release image:

    1. $ CCO_IMAGE=$(oc adm release info --image-for='cloud-credential-operator' $RELEASE_IMAGE)

    Ensure that the architecture of the $RELEASE_IMAGE matches the architecture of the environment in which you will use the ccoctl tool.

  2. Extract the ccoctl binary from the CCO container image within the OKD release image:

    1. $ oc image extract $CCO_IMAGE --file="/usr/bin/ccoctl" -a ~/.pull-secret

  3. Change the permissions to make ccoctl executable:

    1. $ chmod 775 ccoctl

Verification

  • To verify that ccoctl is ready to use, display the help file:

    1. $ ccoctl aws --help

    Output of ccoctl aws --help:

    ``` Creating/updating/deleting cloud credentials objects for AWS cloud

  1. Usage:
  2.   ccoctl aws [command]
  5. Available Commands:
  6.   create-all               Create all the required credentials objects
  7.   create-iam-roles         Create IAM roles
  8.   create-identity-provider Create IAM identity provider
  9.   create-key-pair          Create a key pair
  10.   delete                   Delete credentials objects
  12. Flags:
  13.   -h, --help   help for aws
  15. Use "ccoctl aws [command] --help" for more information about a command.
  16. ```

Creating AWS resources with the Cloud Credential Operator utility

You can use the CCO utility (ccoctl) to create the required AWS resources individually, or with a single command.

Creating AWS resources individually

If you need to review the JSON files that the ccoctl tool creates before modifying AWS resources, or if the process the ccoctl tool uses to create AWS resources automatically does not meet the requirements of your organization, you can create the AWS resources individually. For example, this option might be useful for an organization that shares the responsibility for creating these resources among different users or departments.

Otherwise, you can use the ccoctl aws create-all command to create the AWS resources automatically.

By default, ccoctl creates objects in the directory in which the commands are run. To specify a directory, use the —output-dir flag. This procedure uses <path_to_ccoctl_output_dir> to refer to this location.

Some ccoctl commands make AWS API calls to create or modify AWS resources. To place JSON files on the local file system instead, use the —dry-run flag. These JSON files can be reviewed or modified and then applied with the AWS CLI tool using the —cli-input-json parameters.

Prerequisites

  • Extract and prepare the ccoctl binary.

Procedure

  1. Generate the public and private RSA key files that are used to set up the OpenID Connect provider for the cluster:

    1. $ ccoctl aws create-key-pair

    Example output:

    1. 2021/04/13 11:01:02 Generating RSA keypair
    2. 2021/04/13 11:01:03 Writing private key to /<path_to_ccoctl_output_dir>/serviceaccount-signer.private
    3. 2021/04/13 11:01:03 Writing public key to /<path_to_ccoctl_output_dir>/serviceaccount-signer.public
    4. 2021/04/13 11:01:03 Copying signing key for use by installer

    where serviceaccount-signer.private and serviceaccount-signer.public are the generated key files.

    This command also creates a private key that the cluster requires during installation in /*<path_to_ccoctl_output_dir>*/tls/bound-service-account-signing-key.key.

  2. Create an OpenID Connect identity provider and S3 bucket on AWS:

    1. $ ccoctl aws create-identity-provider --name=<name> --region=<aws_region> --public-key-file=<path_to_ccoctl_output_dir>/serviceaccount-signer.public

    where:

    • *<name>* is the name used to tag any cloud resources that are created for tracking.

    • *<aws-region>* is the AWS region in which cloud resources will be created.

    • *<path_to_ccoctl_output_dir>* is the path to the public key file that the ccoctl aws create-key-pair command generated.

      Example output:

      1. 2021/04/13 11:16:09 Bucket <name>-oidc created
      2. 2021/04/13 11:16:10 OpenID Connect discovery document in the S3 bucket <name>-oidc at .well-known/openid-configuration updated
      3. 2021/04/13 11:16:10 Reading public key
      4. 2021/04/13 11:16:10 JSON web key set (JWKS) in the S3 bucket <name>-oidc at keys.json updated
      5. 2021/04/13 11:16:18 Identity Provider created with ARN: arn:aws:iam::<aws_account_id>:oidc-provider/<name>-oidc.s3.<aws_region>.amazonaws.com

      where 02-openid-configuration is a discovery document and 03-keys.json is a JSON web key set file.

      This command also creates a YAML configuration file in /*<path_to_ccoctl_output_dir>*/manifests/cluster-authentication-02-config.yaml. This file sets the issuer URL field for the service account tokens that the cluster generates, so that the AWS IAM identity provider trusts the tokens.

  3. Create IAM roles for each component in the cluster.

    1. Extract the list of CredentialsRequest objects from the OKD release image:

      1. $ oc adm release extract --credentials-requests --cloud=aws --to=<path_to_directory_with_list_of_credentials_requests>/credrequests quay.io/<path_to>/ocp-release:<version>

    2. Use the ccoctl tool to process all CredentialsRequest objects in the credrequests directory:

      1. $ ccoctl aws create-iam-roles --name=<name> --region=<aws_region> --credentials-requests-dir=<path_to_directory_with_list_of_credentials_requests>/credrequests --identity-provider-arn=arn:aws:iam::<aws_account_id>:oidc-provider/<name>-oidc.s3.<aws_region>.amazonaws.com

      For AWS environments that use alternative IAM API endpoints, such as GovCloud, you must also specify your region with the —region parameter.

      For each CredentialsRequest object, ccoctl creates an IAM role with a trust policy that is tied to the specified OIDC identity provider, and a permissions policy as defined in each CredentialsRequest object from the OKD release image.

Verification

  • To verify that the OKD secrets are created, list the files in the *<path_to_ccoctl_output_dir>*/manifests directory:

    1. $ ll <path_to_ccoctl_output_dir>/manifests

    Example output:

    1. total 24
    2. -rw-------. 1 <user> <user> 161 Apr 13 11:42 cluster-authentication-02-config.yaml
    3. -rw-------. 1 <user> <user> 379 Apr 13 11:59 openshift-cloud-credential-operator-cloud-credential-operator-iam-ro-creds-credentials.yaml
    4. -rw-------. 1 <user> <user> 353 Apr 13 11:59 openshift-cluster-csi-drivers-ebs-cloud-credentials-credentials.yaml
    5. -rw-------. 1 <user> <user> 355 Apr 13 11:59 openshift-image-registry-installer-cloud-credentials-credentials.yaml
    6. -rw-------. 1 <user> <user> 339 Apr 13 11:59 openshift-ingress-operator-cloud-credentials-credentials.yaml
    7. -rw-------. 1 <user> <user> 337 Apr 13 11:59 openshift-machine-api-aws-cloud-credentials-credentials.yaml

You can verify that the IAM roles are created by querying AWS. For more information, refer to AWS documentation on listing IAM roles.

Creating AWS resources with a single command

If you do not need to review the JSON files that the ccoctl tool creates before modifying AWS resources, and if the process the ccoctl tool uses to create AWS resources automatically meets the requirements of your organization, you can use the ccoctl aws create-all command to automate the creation of AWS resources.

Otherwise, you can create the AWS resources individually.

By default, ccoctl creates objects in the directory in which the commands are run. To specify a directory, use the —output-dir flag. This procedure uses <path_to_ccoctl_output_dir> to refer to this location.

Prerequisites

  • Extract and prepare the ccoctl binary.

Procedure

  1. Extract the list of CredentialsRequest objects from the OKD release image:

    1. $ oc adm release extract --credentials-requests --cloud=aws --to=<path_to_directory_with_list_of_credentials_requests>/credrequests quay.io/<path_to>/ocp-release:<version>

  2. Use the ccoctl tool to process all CredentialsRequest objects in the credrequests directory:

    1. $ ccoctl aws create-all --name=<name> --region=<aws_region> --credentials-requests-dir=<path_to_directory_with_list_of_credentials_requests>/credrequests

Verification

  • To verify that the OKD secrets are created, list the files in the *<path_to_ccoctl_output_dir>*/manifests directory:

    1. $ ll <path_to_ccoctl_output_dir>/manifests

    Example output:

    1. total 24
    2. -rw-------. 1 <user> <user> 161 Apr 13 11:42 cluster-authentication-02-config.yaml
    3. -rw-------. 1 <user> <user> 379 Apr 13 11:59 openshift-cloud-credential-operator-cloud-credential-operator-iam-ro-creds-credentials.yaml
    4. -rw-------. 1 <user> <user> 353 Apr 13 11:59 openshift-cluster-csi-drivers-ebs-cloud-credentials-credentials.yaml
    5. -rw-------. 1 <user> <user> 355 Apr 13 11:59 openshift-image-registry-installer-cloud-credentials-credentials.yaml
    6. -rw-------. 1 <user> <user> 339 Apr 13 11:59 openshift-ingress-operator-cloud-credentials-credentials.yaml
    7. -rw-------. 1 <user> <user> 337 Apr 13 11:59 openshift-machine-api-aws-cloud-credentials-credentials.yaml

You can verify that the IAM roles are created by querying AWS. For more information, refer to AWS documentation on listing IAM roles.

Running the installer

Prerequisites

  • Obtain the OKD release image.

Procedure

  1. Change to the directory that contains the installation program and create the install-config.yaml file:

    1. $ openshift-install create install-config --dir <installation_directory>

    where <installation_directory> is the directory in which the installation program creates files.

  2. Edit the install-config.yaml configuration file so that it contains the credentialsMode parameter set to Manual.

    Example install-config.yaml configuration file

    1. apiVersion: v1
    2. baseDomain: cluster1.example.com
    3. credentialsMode: Manual (1)
    4. compute:
    5. - architecture: amd64
    6.   hyperthreading: Enabled
    7. ...
    1This line is added to set the credentialsMode parameter to Manual.

  3. Create the required OKD installation manifests:

    1. $ openshift-install create manifests

  4. Copy the manifests that ccoctl generated to the manifests directory that the installation program created:

    1. $ cp /<path_to_ccoctl_output_dir>/manifests/* ./manifests/

  5. Copy the private key that the ccoctl generated in the tls directory to the installation directory:

    1. $ cp -a /<path_to_ccoctl_output_dir>/tls .

  6. Run the OKD installer:

    1. $ ./openshift-install create cluster

Verifying the installation

  1. Connect to the OKD cluster.

  2. Verify that the cluster does not have root credentials:

    1. $ oc get secrets -n kube-system aws-creds

    The output should look similar to:

    1. Error from server (NotFound): secrets "aws-creds" not found

  3. Verify that the components are assuming the IAM roles that are specified in the secret manifests, instead of using credentials that are created by the CCO:

    Example command with the Image Registry Operator

    1. $ oc get secrets -n openshift-image-registry installer-cloud-credentials -o json | jq -r .data.credentials | base64 --decode

    The output should show the role and web identity token that are used by the component and look similar to:

    Example output with the Image Registry Operator

    1. [default]
    2. role_arn = arn:aws:iam::123456789:role/openshift-image-registry-installer-cloud-credentials
    3. web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token

Upgrading an OKD cluster configured for manual mode with STS

The release image for the version of OKD that you are upgrading to contains a version of the ccoctl binary and list of CredentialsRequest objects specific to that release.

Configuring the Cloud Credential Operator utility

To create and manage cloud credentials from outside of the cluster when the Cloud Credential Operator (CCO) is operating in manual mode with STS, extract and prepare the CCO utility (ccoctl) binary.

The ccoctl is a Linux binary that must run in a Linux environment.

Prerequisites

  • Obtain the OKD release image.

Procedure

  1. Get the CCO container image from the OKD release image:

    1. $ CCO_IMAGE=$(oc adm release info --image-for='cloud-credential-operator' $RELEASE_IMAGE)

    Ensure that the architecture of the $RELEASE_IMAGE matches the architecture of the environment in which you will use the ccoctl tool.

  2. Extract the ccoctl binary from the CCO container image within the OKD release image:

    1. $ oc image extract $CCO_IMAGE --file="/usr/bin/ccoctl" -a ~/.pull-secret

  3. Change the permissions to make ccoctl executable:

    1. $ chmod 775 ccoctl

Verification

  • To verify that ccoctl is ready to use, display the help file:

    1. $ ccoctl aws --help

    Output of ccoctl aws --help:

    ``` Creating/updating/deleting cloud credentials objects for AWS cloud

  1. Usage:
  2.   ccoctl aws [command]
  5. Available Commands:
  6.   create-all               Create all the required credentials objects
  7.   create-iam-roles         Create IAM roles
  8.   create-identity-provider Create IAM identity provider
  9.   create-key-pair          Create a key pair
  10.   delete                   Delete credentials objects
  12. Flags:
  13.   -h, --help   help for aws
  15. Use "ccoctl aws [command] --help" for more information about a command.
  16. ```

Updating AWS resources with the Cloud Credential Operator utility

The process for upgrading an OKD cluster configured for manual mode with AWS Secure Token Service (STS) is similar to installing on a cluster for which you create the AWS resources individually.

By default, ccoctl creates objects in the directory in which the commands are run. To specify a directory, use the —output-dir flag. This procedure uses <path_to_ccoctl_output_dir> to refer to this location.

Some ccoctl commands make AWS API calls to create or modify AWS resources. To place JSON files on the local file system instead, use the —dry-run flag. These JSON files can be reviewed or modified and then applied with the AWS CLI tool using the —cli-input-json parameters.

Prerequisites

  • Obtain the OKD release image for the version that you are upgrading to.

  • Extract and prepare the ccoctl binary from the release image.

Procedure

  1. Extract the list of CredentialsRequest custom resources (CRs) from the OKD release image:

    1. $ oc adm release extract --credentials-requests --cloud=aws --to=<path_to_directory_with_list_of_credentials_requests>/credrequests quay.io/<path_to>/ocp-release:<version>

  2. For each CredentialsRequest CR in the release image, ensure that a namespace that matches the text in the spec.secretRef.namespace field exists in the cluster. This field is where the generated secrets that hold the credentials configuration are stored.

    Sample AWS CredentialsRequest object

    1. apiVersion: cloudcredential.openshift.io/v1
    2. kind: CredentialsRequest
    3. metadata:
    4.   name: cloud-credential-operator-iam-ro
    5.   namespace: openshift-cloud-credential-operator
    6. spec:
    7.   secretRef:
    8.     name: cloud-credential-operator-iam-ro-creds
    9.     namespace: openshift-cloud-credential-operator (1)
    10.   providerSpec:
    11.     apiVersion: cloudcredential.openshift.io/v1
    12.     kind: AWSProviderSpec
    13.     statementEntries:
    14.     - effect: Allow
    15.       action:
    16.       - iam:GetUser
    17.       - iam:GetUserPolicy
    18.       - iam:ListAccessKeys
    19.       resource: "*"
    1This field indicates the namespace which needs to exist to hold the generated secret.

  3. For any CredentialsRequest CR for which the cluster does not already have a namespace with the name specified in spec.secretRef.namespace, create the namespace:

    1. $ oc create namespace <component_namespace>

  4. Use the ccoctl tool to process all CredentialsRequest objects in the credrequests directory:

    1. $ ccoctl aws create-iam-roles --name <name> --region=<aws_region> --credentials-requests-dir=<path_to_directory_with_list_of_credentials_requests>/credrequests --identity-provider-arn arn:aws:iam::<aws_account_id>:oidc-provider/<name>-oidc.s3.<aws_region>.amazonaws.com

    where:

    • <name> is the name used to tag any cloud resources that are created for tracking. For upgrades, use the same value that was used for the initial installation.

    • <aws_account_id> is the AWS account ID.

    • <aws_region> is the AWS region in which cloud resources will be created.

      For AWS environments that use alternative IAM API endpoints, such as GovCloud, you must also specify your region with the —region parameter.

      For each CredentialsRequest object, ccoctl creates an IAM role with a trust policy that is tied to the specified OIDC identity provider, and a permissions policy as defined in each CredentialsRequest object from the OKD release image.

  5. Apply the secrets to your cluster:

    1. $ ls <path_to_ccoctl_output_dir>/manifests/*-credentials.yaml | xargs -I{} oc apply -f {}

Verification

You can verify that the IAM roles are created by querying AWS. For more information, refer to AWS documentation on listing IAM roles.

Upgrading clusters with manually maintained credentials

The Cloud Credential Operator (CCO) Upgradable status for a cluster with manually maintained credentials is False by default.

  • For minor releases, for example, from 4.8 to 4.9, this status prevents you from upgrading until you have addressed any updated permissions and annotated the CloudCredential resource to indicate that the permissions are updated as needed for the next version. This annotation changes the Upgradable status to True.

  • For z-stream releases, for example, from 4.9.0 to 4.9.1, no permissions are added or changed, so the upgrade is not blocked.

Before upgrading a cluster with manually maintained credentials, you must create any new credentials for the release image that you are upgrading to. Additionally, you must review the required permissions for existing credentials and accommodate any new permissions requirements in the new release for those components.

Procedure

  1. Extract and examine the CredentialsRequest custom resource for the new release.

    The “Manually creating IAM” section of the installation content for your cloud provider explains how to obtain and use the credentials required for your cloud.

  2. Update the manually maintained credentials on your cluster:

    • Create new secrets for any CredentialsRequest custom resources that are added by the new release image.

    • If the CredentialsRequest custom resources for any existing credentials that are stored in secrets have changed their permissions requirements, update the permissions as required.

  3. When all of the secrets are correct for the new release, indicate that the cluster is ready to upgrade:

    1. Log in to the OKD CLI as a user with the cluster-admin role.

    2. Edit the CloudCredential resource to add an upgradeable-to annotation within the metadata field:

      1. $ oc edit cloudcredential cluster

      Text to add

      1. ...
      2.   metadata:
      3.     annotations:
      4.       cloudcredential.openshift.io/upgradeable-to: <version_number>
      5. ...

      Where <version_number> is the version you are upgrading to, in the format x.y.z. For example, 4.8.2 for OKD 4.8.2.

      It may take several minutes after adding the annotation for the upgradeable status to change.

  4. Verify that the CCO is upgradeable:

    1. In the Administrator perspective of the web console, navigate to Administration → Cluster Settings.

    2. To view the CCO status details, click cloud-credential in the Cluster Operators list.

    3. If the Upgradeable status in the Conditions section is False, verify that the upgradeable-to annotation is free of typographical errors.

When the Upgradeable status in the Conditions section is True, you can begin the OKD upgrade.

Running the installer

Prerequisites

  • Obtain the OKD release image.

Procedure

  1. Change to the directory that contains the installation program and create the install-config.yaml file:

    1. $ openshift-install create install-config --dir <installation_directory>

    where <installation_directory> is the directory in which the installation program creates files.

  2. Edit the install-config.yaml configuration file so that it contains the credentialsMode parameter set to Manual.

    Example install-config.yaml configuration file

    1. apiVersion: v1
    2. baseDomain: cluster1.example.com
    3. credentialsMode: Manual (1)
    4. compute:
    5. - architecture: amd64
    6.   hyperthreading: Enabled
    7. ...
    1This line is added to set the credentialsMode parameter to Manual.

  3. Create the required OKD installation manifests:

    1. $ openshift-install create manifests

  4. Copy the manifests that ccoctl generated to the manifests directory that the installation program created:

    1. $ cp /<path_to_ccoctl_output_dir>/manifests/* ./manifests/

  5. Copy the private key that the ccoctl generated in the tls directory to the installation directory:

    1. $ cp -a /<path_to_ccoctl_output_dir>/tls .

  6. Run the OKD installer:

    1. $ ./openshift-install create cluster

Verifying the installation

  1. Connect to the OKD cluster.

  2. Verify that the cluster does not have root credentials:

    1. $ oc get secrets -n kube-system aws-creds

    The output should look similar to:

    1. Error from server (NotFound): secrets "aws-creds" not found

  3. Verify that the components are assuming the IAM roles that are specified in the secret manifests, instead of using credentials that are created by the CCO:

    Example command with the Image Registry Operator

    1. $ oc get secrets -n openshift-image-registry installer-cloud-credentials -o json | jq -r .data.credentials | base64 --decode

    The output should show the role and web identity token that are used by the component and look similar to:

    Example output with the Image Registry Operator

    1. [default]
    2. role_arn = arn:aws:iam::123456789:role/openshift-image-registry-installer-cloud-credentials
    3. web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token