Using manual mode with GCP Workload Identity

Manual mode with GCP Workload Identity is supported for Google Cloud Platform (GCP).

In manual mode with GCP Workload Identity, the individual OKD cluster components can impersonate IAM service accounts using short-term, limited-privilege credentials.

Requests for new and refreshed credentials are automated by using an appropriately configured OpenID Connect (OIDC) identity provider, combined with IAM service accounts. OKD signs service account tokens that are trusted by GCP, and can be projected into a pod and used for authentication. Tokens are refreshed after one hour by default.

Using manual mode with GCP Workload Identity changes the content of the GCP credentials that are provided to individual OKD components.

GCP secret format

  1. apiVersion: v1
  2. kind: Secret
  3. metadata:
  4.   namespace: <target_namespace> (1)
  5.   name: <target_secret_name> (2)
  6. data:
  7.   service_account.json: <service_account> (3)
1The namespace for the component.
2The name of the component secret.
3The Base64 encoded service account.

Content of the Base64 encoded service_account.json file using long-lived credentials

  1. {
  2.    "type": "service_account", (1)
  3.    "project_id": "<project_id>",
  4.    "private_key_id": "<private_key_id>",
  5.    "private_key": "<private_key>", (2)
  6.    "client_email": "<client_email_address>",
  7.    "client_id": "<client_id>",
  8.    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  9.    "token_uri": "https://oauth2.googleapis.com/token",
  10.    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  11.    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/<client_email_address>"
  12. }
1The credential type is service_account.
2The private RSA key that is used to authenticate to GCP. This key must be kept secure and is not rotated.

Content of the Base64 encoded service_account.json file using GCP Workload Identity

  1. {
  2.    "type": "external_account", (1)
  3.    "audience": "//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/test-pool/providers/test-provider", (2)
  4.    "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
  5.    "token_url": "https://sts.googleapis.com/v1/token",
  6.    "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/<client_email_address>:generateAccessToken", (3)
  7.    "credential_source": {
  8.       "file": "<path_to_token>", (4)
  9.       "format": {
  10.          "type": "text"
  11.       }
  12.    }
  13. }
1The credential type is external_account.
2The target audience is the GCP Workload Identity provider.
3The resource URL of the service account that can be impersonated with these credentials.
4The path to the service account token inside the pod. By convention, this is /var/run/secrets/openshift/serviceaccount/token for OKD components.

In OKD 4.10.8, image registry support for using GCP Workload Identity was removed due to the discovery of an adverse impact to the image registry. To use the image registry on an OKD 4.10.8 cluster that uses Workload Identity, you must configure the image registry to use long-lived credentials instead. This mitigation is planned to last until Workload Identity support for the image registry is restored in a later release.

Installing an OKD cluster configured for manual mode with GCP Workload Identity

To install a cluster that is configured to use the Cloud Credential Operator (CCO) in manual mode with GCP Workload Identity:

  1. Configure the Cloud Credential Operator utility.

  2. Create the required GCP resources.

  3. Run the OKD installer.

  4. Verify that the cluster is using short-lived credentials.

Configuring the Cloud Credential Operator utility

To create and manage cloud credentials from outside of the cluster when the Cloud Credential Operator (CCO) is operating in manual mode with GCP Workload Identity, extract and prepare the CCO utility (ccoctl) binary.

The ccoctl is a Linux binary that must run in a Linux environment.

Procedure

  1. Obtain the OKD release image:

    1. $ RELEASE_IMAGE=$(./openshift-install version | awk '/release image/ {print $3}')

  2. Get the CCO container image from the OKD release image:

    1. $ CCO_IMAGE=$(oc adm release info --image-for='cloud-credential-operator' $RELEASE_IMAGE)

    Ensure that the architecture of the $RELEASE_IMAGE matches the architecture of the environment in which you will use the ccoctl tool.

  3. Extract the ccoctl binary from the CCO container image within the OKD release image:

    1. $ oc image extract $CCO_IMAGE --file="/usr/bin/ccoctl" -a ~/.pull-secret

  4. Change the permissions to make ccoctl executable:

    1. $ chmod 775 ccoctl

Verification

  • To verify that ccoctl is ready to use, display the help file:

    1. $ ccoctl --help

    Output of ccoctl --help:

    1. OpenShift credentials provisioning tool
    3. Usage:
    4.   ccoctl [command]
    6. Available Commands:
    7.   alibabacloud Manage credentials objects for alibaba cloud
    8.   aws          Manage credentials objects for AWS cloud
    9.   gcp          Manage credentials objects for Google cloud
    10.   help         Help about any command
    11.   ibmcloud     Manage credentials objects for IBM Cloud
    13. Flags:
    14.   -h, --help   help for ccoctl
    16. Use "ccoctl [command] --help" for more information about a command.

Creating GCP resources with the Cloud Credential Operator utility

You can use the ccoctl gcp create-all command to automate the creation of GCP resources.

By default, ccoctl creates objects in the directory in which the commands are run. To specify a directory, use the —output-dir flag. This procedure uses <path_to_ccoctl_output_dir> to refer to this location.

Prerequisites

You must have:

  • Extracted and prepared the ccoctl binary.

Procedure

  1. Extract the list of CredentialsRequest objects from the OKD release image:

    1. $ oc adm release extract --credentials-requests --cloud=gcp --to=<path_to_directory_with_list_of_credentials_requests>/credrequests quay.io/<path_to>/ocp-release:<version>

    This command can take a few moments to run.

  2. Use the ccoctl tool to process all CredentialsRequest objects in the credrequests directory:

    1. $ ccoctl gcp create-all --name=<name> --region=<gcp_region> --project=<gcp_project_id> --credentials-requests-dir=<path_to_directory_with_list_of_credentials_requests>/credrequests

    where:

    • <name> is the user-defined name for all created GCP resources used for tracking.

    • <gcp_region> is the GCP region in which cloud resources will be created.

    • <gcp_project_id> is the GCP project ID in which cloud resources will be created.

    • <path_to_directory_with_list_of_credentials_requests>/credrequests is the directory containing the files of CredentialsRequest manifests to create GCP service accounts.

Verification

  • To verify that the OKD secrets are created, list the files in the <path_to_ccoctl_output_dir>/manifests directory:

    1. $ ls <path_to_ccoctl_output_dir>/manifests

You can verify that the IAM service accounts are created by querying GCP. For more information, refer to GCP documentation on listing IAM service accounts.

Configuring the image registry to use long-lived credentials

In OKD 4.10.8, image registry support for using GCP Workload Identity was removed due to the discovery of an adverse impact to the image registry. To use the image registry on an OKD 4.10.8 cluster that uses Workload Identity, you must configure the image registry to use long-lived credentials instead. This mitigation is planned to last until Workload Identity support for the image registry is restored in a later release.

Prerequisites

  • You have created GCP resources with the Cloud Credential Operator utility (ccoctl).

  • You have obtained and initialized the gcloud CLI.

Procedure

  1. Obtain the email address of the IAM service account that the ccoctl created for the OKD image registry by running the following command:

    1. $ gcloud iam service-accounts list --filter="displayName=<name>-openshift-image-registry-gcs"

    where <name> is the user-defined name for all ccoctl-created GCP resources used for tracking.

  2. Create a long-lived credentials key file for the OKD image registry by running the following command:

    1. $ gcloud iam service-accounts keys create <path_to_image_registry_service_account_keyfile> --iam-account=<image_registry_service_account_email>

    where:

    • <path_to_image_registry_service_account_keyfile> is the path to a location you choose in which to store the key file for the image registry’s IAM service account.

    • <image_registry_service_account_email> is the email address of the IAM service account that the ccoctl created for the OKD image registry.

  3. Generate a Base64-encoded string for the image registry key file by running the following command:

    1. $ cat <path_to_image_registry_service_account_keyfile> | base64 -w 0

  4. In the <path_to_ccoctl_output_dir>/manifests directory, locate the OKD image registry secret manifest YAML file, openshift-image-registry-installer-cloud-credentials-credentials.yaml. In this file, replace the contents of the data.service_account.json field with the Base64-encoded key file.

Running the installer

Prerequisites

  • Obtain the OKD release image.

Procedure

  1. Change to the directory that contains the installation program and create the install-config.yaml file:

    1. $ openshift-install create install-config --dir <installation_directory>

    where <installation_directory> is the directory in which the installation program creates files.

  2. Edit the install-config.yaml configuration file so that it contains the credentialsMode parameter set to Manual.

    Example install-config.yaml configuration file

    1. apiVersion: v1
    2. baseDomain: cluster1.example.com
    3. credentialsMode: Manual (1)
    4. compute:
    5. - architecture: amd64
    6.   hyperthreading: Enabled
    1This line is added to set the credentialsMode parameter to Manual.

  3. Create the required OKD installation manifests:

    1. $ openshift-install create manifests

  4. Copy the manifests that ccoctl generated to the manifests directory that the installation program created:

    1. $ cp /<path_to_ccoctl_output_dir>/manifests/* ./manifests/

  5. Copy the private key that the ccoctl generated in the tls directory to the installation directory:

    1. $ cp -a /<path_to_ccoctl_output_dir>/tls .

  6. Run the OKD installer:

    1. $ ./openshift-install create cluster

Verifying the installation

  1. Connect to the OKD cluster.

  2. Verify that the cluster does not have root credentials:

    1. $ oc get secrets -n kube-system gcp-credentials

    The output should look similar to:

    1. Error from server (NotFound): secrets "gcp-credentials" not found

  3. Verify that the components are assuming the service accounts that are specified in the secret manifests, instead of using credentials that are created by the CCO:

    Example command with the Image Registry Operator

    1. $ oc get secrets -n openshift-image-registry installer-cloud-credentials -o json | jq -r '.data."service_account.json"' | base64 -d

    The output should show the role and web identity token that are used by the component and look similar to:

    Example output with the Image Registry Operator

    1. {
    2.    "type": "external_account", (1)
    3.    "audience": "//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/test-pool/providers/test-provider",
    4.    "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
    5.    "token_url": "https://sts.googleapis.com/v1/token",
    6.    "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/<client-email-address>:generateAccessToken", (2)
    7.    "credential_source": {
    8.       "file": "/var/run/secrets/openshift/serviceaccount/token",
    9.       "format": {
    10.          "type": "text"
    11.       }
    12.    }
    13. }
    1The credential type is external_account.
    2The resource URL of the service account used by the Image Registry Operator.