Configuring for Amazon Web Services (AWS)

Overview

OKD can be configured to access an AWS EC2 infrastructure, including using AWS volumes as persistent storage for application data. After you configure AWS, some additional configurations must be completed on the OKD hosts.

Configuring authorization for Amazon Web Services (AWS)

Permissions AWS instances require either IAM account with Programmatic Access using an access and secret key or IAM role assigned to instances at creation to be able to request and manage load balancers and storage in OKD.

The IAM account or IAM role must have the following policy permissions permissions to have full cloud provider functionality.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:DescribeVolume*",
                "ec2:CreateVolume",
                "ec2:CreateTags",
                "ec2:DescribeInstances",
                "ec2:AttachVolume",
                "ec2:DetachVolume",
                "ec2:DeleteVolume",
                "ec2:DescribeSubnets",
                "ec2:CreateSecurityGroup",
                "ec2:DescribeSecurityGroups",
                "ec2:DeleteSecurityGroup",
                "ec2:DescribeRouteTables",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:RevokeSecurityGroupIngress",
                "elasticloadbalancing:DescribeTags",
                "elasticloadbalancing:CreateLoadBalancerListeners",
                "elasticloadbalancing:ConfigureHealthCheck",
                "elasticloadbalancing:DeleteLoadBalancerListeners",
                "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:CreateLoadBalancer",
                "elasticloadbalancing:DeleteLoadBalancer",
                "elasticloadbalancing:ModifyLoadBalancerAttributes",
                "elasticloadbalancing:DescribeLoadBalancerAttributes"
            ],
            "Resource": "*",
            "Effect": "Allow",
            "Sid": "1"
        }
    ]
}

aws iam put-role-policy \
  --role-name openshift-role \
  --policy-name openshift-admin \
  --policy-document file: //openshift_iam_policy

aws iam put-user-policy \
  --user-name openshift-admin \
  --policy-name openshift-admin \
  --policy-document file: //openshift_iam_policy

Configuring the OKD cloud provider at installation

Procedure

To configure the configure the Amazon Web Services cloud provider using an IAM account with an access and secret key add the following values to the inventory:

[OSEv3:vars]
openshift_cloudprovider_kind=aws
openshift_clusterid=openshift (1)
openshift_cloudprovider_aws_access_key=AKIAJ6VLBLISADPBUA (2)
openshift_cloudprovider_aws_secret_key=g/8PmDNYHVSQn0BQE+xtsHzbaZaGYjGNzhbdgwjH (3)

To configure the configure the Amazon Web Services cloud provider using an IAM role add the following values to the inventory:

[source,yaml]
----
[OSEv3:vars]
openshift_cloudprovider_kind=aws
openshift_clusterid=openshift (1)
----
<1> A tag assigned to all resources (instances, load balancers, vpc, etc) used for OpenShift.

NOTE: The IAM role takes the place of needing an access and secret key.

Configuring the OKD cloud provider after installation

In the event that the Amazon Web Services cloud provider values were not provided at installation time the configuration can be defined and created after the installation. Follow the steps to configure the configuration file and manually configuring the master and node Manually Configuring OKD Masters for AWS.

Configuring a Security Group

When installing OKD on AWS, ensure that you set up the appropriate security groups.

These are some ports that you must have in your security groups, without which the installation fails. You may need more depending on the cluster configuration you want to install. For more information and to adjust your security groups accordingly, see Required Ports for more information.

If configuring external load-balancers (ELBs) for load balancing the masters and/or routers, you also need to configure Ingress and Egress security groups for the ELBs appropriately.

Overriding Detected IP Addresses and Host Names

In AWS, situations that require overriding the variables include:

For EC2 hosts in particular, they must be deployed in a VPC that has both **DNS host names** and **DNS resolution** enabled.

Configuring the OKD registry for Amazon Web Services (AWS)

Amazon Web Services (AWS) provides object cloud storage that OKD can use to store container images using the OKD container registry.

For more information, see Amazon S3.

Prerequisites

OKD uses S3 for image storage. A S3 bucket, IAM policy, and IAM user with Programmatic Access should be created to allow for the installer to configure the registry.

The example below uses awscli to create a bucket with the name of openshift-registry-storage in the region of us-east-1.

# aws s3api create-bucket \
     --bucket openshift-registry-storage \
     --region us-east-1

The default policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads"
      ],
      "Resource": "arn:aws:s3:::S3_BUCKET_NAME"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:ListMultipartUploadParts",
        "s3:AbortMultipartUpload"
      ],
      "Resource": "arn:aws:s3:::S3_BUCKET_NAME/*"
    }
  ]
}

Configuring the OKD inventory to use S3

Procedure

To configure the Ansible inventory for the registry to use the S3 bucket and IAM user:

[OSEv3:vars]
# AWS Registry Configuration
openshift_hosted_manage_registry=true
openshift_hosted_registry_storage_kind=object
openshift_hosted_registry_storage_provider=s3
openshift_hosted_registry_storage_s3_accesskey=AKIAJ6VLREDHATSPBUA (1)
openshift_hosted_registry_storage_s3_secretkey=g/8PmTYDQVGssFWWFvfawHpDbZyGkjGNZhbWQpjH (2)
openshift_hosted_registry_storage_s3_bucket=openshift-registry-storage (3)
openshift_hosted_registry_storage_s3_region=us-east-1 (4)
openshift_hosted_registry_storage_s3_chunksize=26214400
openshift_hosted_registry_storage_s3_rootdirectory=/registry
openshift_hosted_registry_storage_s3_encrypt=false
openshift_hosted_registry_storage_s3_kmskeyid=aws_kms_key_id (5)
openshift_hosted_registry_pullthrough=true
openshift_hosted_registry_acceptschema2=true
openshift_hosted_registry_enforcequota=true
openshift_hosted_registry_replicas=3

Manually configuring OKD registry to use S3

To use Amazon Web Services (AWS) S3 object storage, edit the registry’s configuration file and mount to the registry pod.

Procedure

1. Export the current config.yml:

   $ oc get secret registry-config \
       -o jsonpath='{.data.config\.yml}' -n default | base64 -d \
     >> config.yml.old

2. Create a new configuration file from the old config.yml:

   $ cp config.yml.old config.yml

3. Edit the file to include the S3 parameters. Specify the accountname, accountkey, container, and realm in the storage section of a registry’s configuration file:

   storage:
     delete:
       enabled: true
     cache:
       blobdescriptor: inmemory
     s3:
       accesskey: AKIAJ6VLREDHATSPBUA (1)
       secretkey: g/8PmTYDQVGssFWWFvfawHpDbZyGkjGNZhbWQpjH (2)
       region: us-east-1 (3)
       bucket: openshift-registry-storage (4)
       encrypt: False
       secure: true
       v4auth: true
       rootdirectory: /registry (5)
       chunksize: "26214400"

   1	Replace with an AWS access key that is authorized to access the S3 bucket.
   2	The secret key that corresponds to the defined AWS access key.
   3	The name of the S3 bucket to be used as the registry.
   4	The location in which the registry will store images and metadata. (Default is /registry)

4. Delete the registry-config secret:

   $ oc delete secret registry-config -n default

5. Recreate the secret to reference the updated configuration file:

   $ oc create secret generic registry-config \
       --from-file=config.yml -n default

6. Redeploy the registry to read the updated configuration:

   $ oc rollout latest docker-registry -n default

Verify the registry is using S3 storage

To verify if the registry is using Amazon S3 storage:

Procedure

1. After a successful registry deployment, the registry deploymentconfig describes registry-storage as emptydir instead of AWS S3 but the configuration for the AWS S3 bucket resides in the secret docker-config. The docker-config secret mounts to REGISTRY_CONFIGURATION_PATH which provides all of the paramaters when using AWS S3 for the registry object storage.

   $ oc describe dc docker-registry -n default
   ...
       Environment:
         REGISTRY_HTTP_ADDR:                    :5000
         REGISTRY_HTTP_NET:                    tcp
         REGISTRY_HTTP_SECRET:                    SPLR83SDsPaGbGuwSMDfnDwrDRvGf6YXl4h9JQrToQU=
         REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA:    false
         REGISTRY_HTTP_TLS_KEY:                    /etc/secrets/registry.key
         OPENSHIFT_DEFAULT_REGISTRY:                docker-registry.default.svc:5000
         REGISTRY_CONFIGURATION_PATH:                /etc/registry/config.yml
         REGISTRY_OPENSHIFT_SERVER_ADDR:                docker-registry.default.svc:5000
         REGISTRY_HTTP_TLS_CERTIFICATE:                /etc/secrets/registry.crt
       Mounts:
         /etc/registry from docker-config (rw)
         /etc/secrets from registry-certificates (rw)
         /registry from registry-storage (rw)
     Volumes:
      registry-storage:
       Type:    EmptyDir (a temporary directory that shares a pod's lifetime) (1)
       Medium:
      registry-certificates:
       Type:    Secret (a volume populated by a Secret)
       SecretName:    registry-certificates
       Optional:    false
      docker-config:
       Type:    Secret (a volume populated by a Secret)
       SecretName:    registry-config
       Optional:    false
   ....

   1	The temporary directory that shares a pod’s lifetime.

2. Ensure that the /registry mountpoint is empty:

   $ oc exec \
       $(oc get pod -l deploymentconfig=docker-registry \
       -o=jsonpath='{.items[0].metadata.name}')  -i -t -- ls -l /registry
   total 0

   If it is empty, it is because the S3 configuration is defined in the registry-config secret:

   $ oc describe secret registry-config
   Name:         registry-config
   Namespace:    default
   Labels:       <none>
   Annotations:  <none>
   Type:  Opaque
   Data
   ====
   config.yml:  398 bytes

3. The installer creates a config.yml file with the desired configuration using the extended registry capabilities as seen in Storage in the installation documentation. To view the configuration file, including the storage section where the storage bucket configuration is stored:

   $ oc exec \
       $(oc get pod -l deploymentconfig=docker-registry \
         -o=jsonpath='{.items[0].metadata.name}') \
     cat /etc/registry/config.yml
     version: 0.1
     log:
       level: debug
     http:
       addr: :5000
     storage:
       delete:
         enabled: true
       cache:
         blobdescriptor: inmemory
       s3:
         accesskey: AKIAJ6VLREDHATSPBUA
         secretkey: g/8PmTYDQVGssFWWFvfawHpDbZyGkjGNZhbWQpjH
         region: us-east-1
         bucket: openshift-registry-storage
         encrypt: False
         secure: true
         v4auth: true
         rootdirectory: /registry
         chunksize: "26214400"
     auth:
       openshift:
         realm: openshift
     middleware:
       registry:
       - name: openshift
       repository:
       - name: openshift
         options:
           pullthrough: true
           acceptschema2: true
           enforcequota: true
       storage:
       - name: openshift

   Alternatively, you can view the secret:

   $ oc get secret registry-config -o jsonpath='{.data.config\.yml}' | base64 -d
   version: 0.1
   log:
     level: debug
   http:
     addr: :5000
     storage:
       delete:
         enabled: true
       cache:
         blobdescriptor: inmemory
       s3:
         accesskey: AKIAJ6VLREDHATSPBUA
         secretkey: g/8PmTYDQVGssFWWFvfawHpDbZyGkjGNZhbWQpjH
         region: us-east-1
         bucket: openshift-registry-storage
         encrypt: False
         secure: true
         v4auth: true
         rootdirectory: /registry
         chunksize: "26214400"
   auth:
     openshift:
       realm: openshift
   middleware:
     registry:
     - name: openshift
     repository:
     - name: openshift
     options:
       pullthrough: true
       acceptschema2: true
       enforcequota: true
     storage:
     - name: openshift

If using an emptyDir volume, the /registry mountpoint looks like the following:

$ oc exec \
    $(oc get pod -l deploymentconfig=docker-registry \
    -o=jsonpath='{.items[0].metadata.name}')  -i -t -- df -h /registry
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc         100G  226M   30G   1% /registry
$ oc exec \
    $(oc get pod -l deploymentconfig=docker-registry \
    -o=jsonpath='{.items[0].metadata.name}')  -i -t -- ls -l /registry
total 0
drwxr-sr-x. 3 1000000000 1000000000 22 Jun 19 12:24 docker

Configuring AWS Variables

To set the required AWS variables, create a /etc/origin/cloudprovider/aws.conf file with the following contents on all of your OKD hosts, both masters and nodes:

[Global]
Zone = us-east-1c (1)

Configuring OKD for AWS

You can set the AWS configuration on OKD in two ways:

• using Ansible or

• manually, by modifying the master-config.yaml, node-config.yaml, and related /etc/sysconfig/ files.

Configuring OKD for AWS with Ansible

During cluster installations, AWS can be configured using the openshift_cloudprovider_aws_access_key, openshift_cloudprovider_aws_secret_key, openshift_cloudprovider_kind, openshift_clusterid parameters, which are configurable in the inventory file.

Example AWS Configuration with Ansible

# Cloud Provider Configuration
#
# Note: You may make use of environment variables rather than store
# sensitive configuration within the ansible inventory.
# For example:
#openshift_cloudprovider_aws_access_key="{{ lookup('env','AWS_ACCESS_KEY_ID') }}"
#openshift_cloudprovider_aws_secret_key="{{ lookup('env','AWS_SECRET_ACCESS_KEY') }}"
#
#openshift_clusterid=unique_identifier_per_availablility_zone
#
# AWS (Using API Credentials)
#openshift_cloudprovider_kind=aws
#openshift_cloudprovider_aws_access_key=aws_access_key_id
#openshift_cloudprovider_aws_secret_key=aws_secret_access_key
#
# AWS (Using IAM Profiles)
#openshift_cloudprovider_kind=aws
# Note: IAM roles must exist before launching the instances.

Manually Configuring OKD Masters for AWS

Edit or create the master configuration file on all masters (/etc/origin/master/master-config.yaml by default) and update the contents of the **apiServerArguments** and **controllerArguments** sections:

kubernetesMasterConfig:
  ...
  apiServerArguments:
    cloud-provider:
      - "aws"
    cloud-config:
      - "/etc/origin/cloudprovider/aws.conf"
  controllerArguments:
    cloud-provider:
      - "aws"
    cloud-config:
      - "/etc/origin/cloudprovider/aws.conf"

Currently, the nodeName must match the instance name in AWS in order for the cloud provider integration to work properly. The name must also be RFC1123 compliant.

Manually Configuring OKD Nodes for AWS

Edit the appropriate node configuration map and update the contents of the **kubeletArguments** section:

kubeletArguments:
  cloud-provider:
    - "aws"
  cloud-config:
    - "/etc/origin/cloudprovider/aws.conf"

Manually Setting Key-Value Access Pairs

Make sure the following environment variables are set in the /etc/origin/master/master.env file on masters and the /etc/sysconfig/origin-node file on nodes:

AWS_ACCESS_KEY_ID=<key_ID>
AWS_SECRET_ACCESS_KEY=<secret_key>

Applying Configuration Changes

Start or restart OKD services on all master and node hosts to apply your configuration changes, see Restarting OKD services:

# master-restart api
# master-restart controllers
# systemctl restart atomic-openshift-node

Switching from not using a cloud provider to using a cloud provider produces an error message. Adding the cloud provider tries to delete the node because the node switches from using the hostname as the **externalID** (which would have been the case when no cloud provider was being used) to using the cloud provider’s **instance-id** (which is what the cloud provider specifies). To resolve this issue:

1. Log in to the CLI as a cluster administrator.

2. Check and back up existing node labels:

   $ oc describe node <node_name> | grep -Poz '(?s)Labels.*\n.*(?=Taints)'

3. Delete the nodes:

   $ oc delete node <node_name>

4. On each node host, restart the OKD service.

   # systemctl restart origin-node

5. Add back any labels on each node that you previously had.

Labeling Clusters for AWS

If you configure AWS provider credentials, you must also ensure that all hosts are labeled.

To correctly identify which resources are associated with a cluster, tag resources with the key kubernetes.io/cluster/<clusterid>, where:

• <clusterid> is a unique name for the cluster.

Set the corresponding value to owned if the node belongs exclusively to the cluster or to shared if it is a resource shared with other systems.

Tagging all resources with the kubernetes.io/cluster/<clusterid>,Value=(owned|shared) tag avoids potential issues with multiple zones or multiple clusters.

See Pods and Services to learn more about labeling and tagging in OKD.

Resources That Need Tags

There are four types of resources that need to be tagged:

• Instances

• Security Groups

• Load Balancers

• EBS Volumes

Tagging an Existing Cluster

A cluster uses the value of the kubernetes.io/cluster/<clusterid>,Value=(owned|shared) tag to determine which resources belong to the AWS cluster. This means that all relevant resources must be labeled with the kubernetes.io/cluster/<clusterid>,Value=(owned|shared) tag using the same values for that key. These resources include:

• All hosts.

• All relevant load balancers to be used in the AWS instances.

• All EBS volumes. The EBS Volumes that need to be tagged can found with:

  $ oc get pv -o json|jq '.items[].spec.awsElasticBlockStore.volumeID'

• All relevant security groups to be used with the AWS instances.

  Do not tag all existing security groups with the kubernetes.io/cluster/<name>,Value=<clusterid> tag, or the Elastic Load Balancing (ELB) will not be able to create a load balancer.

After tagging any resources, restart the master services on the master and restart the node service on all nodes. See the Applying Configuration Section.

About Red Hat OpenShift Container Storage

Red Hat OpenShift Container Storage (RHOCS) is a provider of agnostic persistent storage for OKD either in-house or in hybrid clouds. As a Red Hat storage solution, RHOCS is completely integrated with OKD for deployment, management, and monitoring regardless if it is installed on OKD (converged) or with OKD (independent). OpenShift Container Storage is not limited to a single availability zone or node, which makes it likely to survive an outage. You can find complete instructions for using RHOCS in the RHOCS3.11 Deployment Guide.