Configuring for OpenStack

Overview

When deployed on OpenStack, OKD can be configured to access the OpenStack infrastructure, including using OpenStack Cinder volumes as persistent storage for application data.

Before you Begin

OKD SDN

The default OKD SDN is OpenShiftSDN. There is another option: use Kuryr SDN.

Kuryr SDN

Kuryr is a CNI plug-in that uses Neutron and Octavia to provide networking for pods and services. It is primarily designed for OKD clusters that run on OpenStack virtual machines. Kuryr improves the network performance by plugging OKD pods into OpenStack SDN. In addition it provides interconnectivity between OKD pods and OpenStack virtual instances.

Kuryr is recommended for OKD deployments on encapsulated OpenStack tenant networks in order to avoid double encapsulation, such as running an encapsulated OpenShift SDN over an OpenStack network. Kuryr is recommended whenever VXLAN, GRE, or GENEVE are required.

Conversely, implementing Kuryr does not make sense in the following cases:

• You use provider networks, tenant VLANs, or a third party commercial SDN such as Cisco ACI or Juniper Contrail.

• The deployment will use many services on a few hypervisors, or OKD virtual machine nodes. Each OKD service creates an Octavia Amphora virtual machine in OpenStack that hosts a required load balancer.

To enable Kuryr SDN, your environment must meet the following requirements:

• Running OpenStack 13 or later

• Overcloud with Octavia

• Neutron Trunk ports extension enabled

• If ML2/OVS Neutron driver is used the OpenvSwitch firewall driver must be used, instead of the ovs-hybrid one.

OKD Prerequisites

A successful deployment of OKD requires many prerequisites. This consists of a set of infrastructure and host configuration steps prior to the actual installation of OKD using Ansible. In the following subsequent sections, details regarding the prerequisites and configuration changes required for an OKD on a OpenStack environment are discussed in detail.

Example:

Enable the rhel-7-server-openstack-13-tools-rpms and the required OKD repositories from Set Up Repositories.

$ sudo subscription-manager repos \
--enable rhel-7-server-openstack-{rhosp_version}-tools-rpms \
--enable rhel-7-server-openstack-14-tools-rpms
$ sudo subscription-manager repo-override --repo=rhel-7-server-openstack-14-tools-rpms --add=includepkgs:"python2-openstacksdk.* python2-keystoneauth1.* python2-os-service-types.*"
$ sudo yum install -y python2-openstackclient python2-heatclient python2-octaviaclient ansible

Verify the packages are of at least the following versions (use rpm -q <package_name>):

• python2-openstackclient - 3.14.1.-1

• python2-heatclient 1.14.0-1

• python2-octaviaclient 1.4.0-1

• python2-openstacksdk 0.17.2

Enabling Octavia: OpenStack Load Balancing as a Service (LBaaS)

Octavia is a supported load balancer solution that is recommended to be used in conjunction with OKD in order to load balance the external incoming traffic and provide a single view of the OKD master services for the applications.

In order to enable Octavia, the Octavia service must be included during the installation of the OpenStack overcloud or upgraded if the overcloud already exists. The following steps provide basic non-custom steps in enabling Octavia and apply to both either a clean install of the overcloud or an overcloud update.

If using the local registry, create a template to upload the images to the registry. Example shown below.

(undercloud) $ openstack overcloud container image prepare \
-e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml \
--namespace=registry.access.redhat.com/rhosp13 \
--push-destination=<local-ip-from-undercloud.conf>:8787 \
--prefix=openstack- \
--tag-from-label {version}-{release} \
--output-env-file=/home/stack/templates/overcloud_images.yaml \
--output-images-file /home/stack/local_registry_images.yaml

Verify that the created local_registry_images.yaml contains the Octavia images.

Octavia images in local registry file

...
- imagename: registry.access.redhat.com/rhosp13/openstack-octavia-api:13.0-43
  push_destination: <local-ip-from-undercloud.conf>:8787
- imagename: registry.access.redhat.com/rhosp13/openstack-octavia-health-manager:13.0-45
  push_destination: <local-ip-from-undercloud.conf>:8787
- imagename: registry.access.redhat.com/rhosp13/openstack-octavia-housekeeping:13.0-45
  push_destination: <local-ip-from-undercloud.conf>:8787
- imagename: registry.access.redhat.com/rhosp13/openstack-octavia-worker:13.0-44
  push_destination: <local-ip-from-undercloud.conf>:8787

The following step pulls the container images from registry.redhat.io to the undercloud node. This process might take some time depending on the speed of the network and undercloud disk.

(undercloud) $ sudo openstack overcloud container image upload \
  --config-file  /home/stack/local_registry_images.yaml \
  --verbose

As an Octavia Load Balancer is used to access the OpenShift API, there is a need to increase their listeners default timeouts for the connections. The default timeout is 50 seconds. Increase the timeout to 20 minutes by passying the following file to the overcloud deploy command:

(undercloud) $ cat octavia_timeouts.yaml
parameter_defaults:
  OctaviaTimeoutClientData: 1200000
  OctaviaTimeoutMemberData: 1200000

Install or update your overcloud environment with Octavia:

openstack overcloud deploy --templates \
.
.
.
  -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml \
  -e octavia_timeouts.yaml
.
.
.

Creating OpenStack User Accounts, Projects, and Roles

Before installing OKD, the Red Hat OpenStack Platform (RHOSP) environment requires a project, often referred to as a tenant, that stores the OpenStack instances that are to install the OKD. This project requires ownership by a user and the role of that user to be set to _member_.

The following steps show how to accomplish the above.

As the OpenStack overcloud administrator,

1. Create a project (tenant) that is to store the RHOSP instances

   $ openstack project create <project>

2. Create a RHOSP user that has ownership of the previously created project:

   $ openstack user create --password <password> <username>

3. Set the role of the user:

   $ openstack role add --user <username> --project <project> _member_

The default quotas assigned to new RH OSP projects are not high enough for OKD installations. Increase the quotas to at least 30 security groups, 200 security group rules, and 200 ports.

$ openstack quota set --secgroups 30 --secgroup-rules 200 --ports 200 <project>
(1)

Extra steps for Kuryr SDN

If Kuryr SDN is enabled, especially if you are using namespace isolation, increase your project’s quotas to meet these minimum requirements:

• 300 security groups - one for each namespace plus one for each load balancer

• 150 networks - one for each namespace

• 150 subnets - one for each namespace

• 500 security group rules

• 500 ports - one port per Pod and additional ports for pools to speed up Pod creation

If you are using namespace isolation, each namespace is given a new network and subnet. Additionally, a security group is created to enable traffic between Pods in the namespace.

$ openstack quota set --networks 150 --subnets 150 --secgroups 300 --secgroup-rules 500 --ports 500 <project>
(1)

If you enabled namespace isolation, you must add the project ID to the octavia.conf configuration file after you create the project. This step ensures that required LoadBalancer security groups belong to that project and that they can be updated to enforce services isolation across namespaces.

1. Get the project ID

   $ openstack project show *<project>*
   +-------------+----------------------------------+
   | Field       | Value                            |
   +-------------+----------------------------------+
   | description |                                  |
   | domain_id   | default                          |
   | enabled     | True                             |
   | id          | PROJECT_ID                       |
   | is_domain   | False                            |
   | name        | *<project>*                      |
   | parent_id   | default                          |
   | tags        | []                               |
   +-------------+----------------------------------+

2. Add the project ID to [filename]octavia.conf on the controllers and restart octavia worker.

   $ source stackrc  # undercloud credentials
   $ openstack server list
   +--------------------------------------+--------------+--------+-----------------------+----------------+------------+
   │
   | ID                                   | Name         | Status | Networks
   | Image          | Flavor     |
   │
   +--------------------------------------+--------------+--------+-----------------------+----------------+------------+
   │
   | 6bef8e73-2ba5-4860-a0b1-3937f8ca7e01 | controller-0 | ACTIVE |
   ctlplane=192.168.24.8 | overcloud-full | controller |
   │
   | dda3173a-ab26-47f8-a2dc-8473b4a67ab9 | compute-0    | ACTIVE |
   ctlplane=192.168.24.6 | overcloud-full | compute    |
   │
   +--------------------------------------+--------------+--------+-----------------------+----------------+------------+
   $ ssh heat-admin@192.168.24.8  # ssh into the controller(s)
   controller-0$ vi /var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf
   [controller_worker]
   # List of project ids that are allowed to have Load balancer security groups
   # belonging to them.
   amp_secgroup_allowed_projects = PROJECT_ID
   controller-0$ sudo docker restart octavia_worker

Configuring the RC file

After you configure the project, an OpenStack administrator can create an RC file with all the required information to the user(s) implementing the OKD environment.

An example RC file:

$ cat path/to/examplerc
# Clear any old environment that may conflict.
for key in $( set | awk '{FS="="}  /^OS_/ {print $1}' ); do unset $key ; done
export OS_PROJECT_DOMAIN_NAME=Default
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_NAME=<project-name>
export OS_USERNAME=<username>
export OS_PASSWORD=<password>
export OS_AUTH_URL=http://<ip>:5000//v3
export OS_CLOUDNAME=<cloud-name>
export OS_IDENTITY_API_VERSION=3
# Add OS_CLOUDNAME to PS1
if [ -z "${CLOUDPROMPT_ENABLED:-}" ]; then
    export PS1=${PS1:-""}
    export PS1=\${OS_CLOUDNAME:+"(\$OS_CLOUDNAME)"}\ $PS1
    export CLOUDPROMPT_ENABLED=1
fi

As the user(s) implementing the OKD environment, within the OpenStack director node or workstation, ensure to source the credentials as follows:

$ source path/to/examplerc

Create an OpenStack Flavor

Within OpenStack, flavors define the size of a virtual server by defining the compute, memory, and storage capacity of nova computing instances. Since the base image within this reference architecture is Red Hat Enterprise Linux 7.5, a m1.node and m1.master sized flavor is created with the following specifications as shown in Minimum System Requirements for OpenShift.

As an OpenStack administrator,

$ openstack flavor create <flavor_name> \
    --id auto \
    --ram <ram_in_MB> \
    --disk <disk_in_GB> \
    --vcpus <num_vcpus>

An example below showing the creation of flavors within this reference environment.

$ openstack flavor create m1.master \
    --id auto \
    --ram 16384 \
    --disk 45 \
    --vcpus 4
$ openstack flavor create m1.node \
    --id auto \
    --ram 8192 \
    --disk 20 \
    --vcpus 1

Verification of the OpenStack flavors via:

$ openstack flavor list

Creating an OpenStack Keypair

Red Hat OpenStack Platform uses cloud-init to place an ssh public key on each instance as it is created to allow ssh access to the instance. Red Hat OpenStack Platform expects the user to hold the private key.

To generate a keypair, use the following command:

$ openstack keypair create <keypair-name> > /path/to/<keypair-name>.pem

Verification of the keypair creation can be done via:

$ openstack keypair list

Once the keypair is created, set the permissions to 600 thus only allowing the owner of the file to read and write to that file.

$ chmod 600 /path/to/<keypair-name>.pem

Setting up DNS for OKD

DNS service is an important component in the OKD environment. Regardless of the provider of DNS, an organization is required to have certain records in place to serve the various OKD components.

Using the key secret of the DNS, you can provide the information to the OpenShift Ansible Installer and it will automatically add A records for the target instances and the various OKD components. This process setup is described later when configuring the OpenShift Ansible Installer.

Access to a DNS server is expected. You can use Red Hat Labs DNS Helper for assistance with access.

Application DNS

Applications served by OpenShift are accessible by the router on ports 80/TCP and 443/TCP. The router uses a wildcard record to map all host names under a specific sub domain to the same IP address without requiring a separate record for each name.

This allows OKD to add applications with arbitrary names as long as they are under that sub domain.

For example, a wildcard record for *.apps.example.com causes DNS name lookups for tax.apps.example.com and home-goods.apps.example.com to both return the same IP address: 10.19.x.y. All traffic is forwarded to the OpenShift Routers. The Routers examine the HTTP headers of the queries and forward them to the correct destination.

With a load-balancer such as Octavia, host address of 10.19.x.y, the wildcard DNS record can be added as follows:

Creation of OKD Networks via OpenStack

When deploying OKD on Red Hat OpenStack Platform as described in this segment, the requirements are two networks — public and internal network.

Public Network

The public network is a network that contains external access and can be reached by the outside world. The public network creation can be only done by an OpenStack administrator.

The following commands provide an example of creating an OpenStack provider network for public network access.

As an OpenStack administrator (overcloudrc access),

$ source /path/to/examplerc
$ openstack network create <public-net-name> \
  --external \
  --provider-network-type flat \
  --provider-physical-network datacentre
$ openstack subnet create <public-subnet-name> \
  --network <public-net-name> \
  --dhcp \
  --allocation-pool start=<float_start_ip>,end=<float_end_ip> \
  --gateway <ip> \
  --subnet-range <CIDR>

Once the network and subnet have been created verify via:

$ openstack network list
$ openstack subnet list

Internal Network

The internal network is connected to the public network via a router during the network setup. This allows each Red Hat OpenStack Platform instance attached to the internal network the ability to request a floating IP from the public network for public access. The internal network is created automically by the OpenShift Ansible installer via setting the openshift_openstack_private_network_name. More information regarding changes required for the OpenShift Ansible installer are described later.

Creating OpenStack Deployment Host Security Group

OpenStack networking allows the user to define inbound and outbound traffic filters that can be applied to each instance on a network. This allows the user to limit network traffic to each instance based on the function of the instance services and not depend on host based filtering. The OpenShift Ansible installer handles the proper creation of all the ports and services required for each type of host that is part of the OKD cluster except for the deployment host.

The following command creates an empty security group with no rules set for the deployment host.

$ source path/to/examplerc
$ openstack security group create <deployment-sg-name>

Verify the creation of the security group:

$ openstack security group list

Deployment Host Security Group

The deployment instance only needs to allow inbound ssh. This instance exists to give operators a stable base to deploy, monitor and manage the OKD environment.

Creation of the above security group rules is as follows:

$ source /path/to/examplerc
$ openstack security group rule create \
    --ingress \
    --protocol icmp \
    <deployment-sg-name>
$ openstack security group rule create \
    --ingress \
    --protocol tcp \
    --dst-port 22 \
    <deployment-sg-name>

Verification of the security group rules is as follows:

$ openstack security group rule list <deployment-sg-name>
+--------------------------------------+-------------+-----------+------------+-----------------------+
| ID                                   | IP Protocol | IP Range  | Port Range | Remote Security Group |
+--------------------------------------+-------------+-----------+------------+-----------------------+
| 7971fc03-4bfe-4153-8bde-5ae0f93e94a8 | icmp        | 0.0.0.0/0 |            | None                  |
| b8508884-e82b-4ee3-9f36-f57e1803e4a4 | None        | None      |            | None                  |
| cb914caf-3e84-48e2-8a01-c23e61855bf6 | tcp         | 0.0.0.0/0 | 22:22      | None                  |
| e8764c02-526e-453f-b978-c5ea757c3ac5 | None        | None      |            | None                  |
+--------------------------------------+-------------+-----------+------------+-----------------------+

OpenStack Cinder Volumes

OpenStack Block Storage provides persistent block storage management via the cinder service. Block storage enables the OpenStack user to create a volume that may be attached to different OpenStack instances.

Docker Volume

The master and node instances contain a volume to store docker images. The purpose of the volume is to ensure that a large image or container does not compromise node performance or abilities of the existing node.

The docker volume is created by the OpenShift Ansible installer via the variable openshift_openstack_docker_volume_size. More information regarding changes required for the OpenShift Ansible installer are described later.

Registry volume

The OpenShift image registry requires a cinder volume to ensure that images are saved in the event that the registry needs to migrate to another node. The following steps show how to create the image registry via OpenStack. Once the volume is created, the volume ID will be included in the OpenShift Ansible Installer OSEv3.yml file via the parameter openshift_hosted_registry_storage_openstack_volumeID as described later.

$ source /path/to/examplerc
$ openstack volume create --size <volume-size-in-GB> <registry-name>

Verify the creation of the volume.

$ openstack volume list
----------------------------------------+------------------------------------------------+
| ID                                   | Name          | Status    | Size | Attached to  |
+--------------------------------------+-------------------------------------------------+
| d65209f0-9061-4cd8-8827-ae6e2253a18d | <registry-name>| available |   30 |              |
+--------------------------------------+-------------------------------------------------+

Creating and Configuring the Deployment Instance

The role of the deployment instance is to serve as a utility host for the deployment and management of OKD.

Creating the Deployment Host Network and Router

Prior to instance creation, an internal network and router must be created for communication with the deployment host. The following commands create that network and router.

$ source path/to/examplerc
$ openstack network create <deployment-net-name>
$ openstack subnet create --network <deployment-net-name> \
  --subnet-range <subnet_range> \
  --dns-nameserver <dns-ip> \
  <deployment-subnet-name>
$ openstack router create <deployment-router-name>
$ openstack router set --external-gateway <public-net-name> <deployment-router-name>
$ openstack router add subnet <deployment-router-name> <deployment-subnet-name>

Deploying the Deployment Instance

With the network and security group created, deploy the instance.

$ domain=<domain>
$ netid1=$(openstack network show <deployment-net-name> -f value -c id)
$ openstack server create \
    --nic net-id=$netid1 \
    --flavor <flavor> \
    --image <image> \
    --key-name <keypair> \
    --security-group <deployment-sg-name> \
    deployment.$domain

Creating and Adding Floating IP to the Deployment Instance

Once the deployment instance is created, a floating IP must be created and then allocated to the instance. The following shows an example.

$ source /path/to/examplerc
$ openstack floating ip create <public-network-name>
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| created_at          | 2017-08-24T22:44:03Z                 |
| description         |                                      |
| fixed_ip_address    | None                                 |
| floating_ip_address | 10.20.120.150                       |
| floating_network_id | 084884f9-d9d2-477a-bae7-26dbb4ff1873 |
| headers             |                                      |
| id                  | 2bc06e39-1efb-453e-8642-39f910ac8fd1 |
| port_id             | None                                 |
| project_id          | ca304dfee9a04597b16d253efd0e2332     |
| project_id          | ca304dfee9a04597b16d253efd0e2332     |
| revision_number     | 1                                    |
| router_id           | None                                 |
| status              | DOWN                                 |
| updated_at          | 2017-08-24T22:44:03Z                 |
+---------------------+--------------------------------------+

Within the above output, the floating_ip_address field shows that the floating IP 10.20.120.150 is created. In order to assign this IP to the deployment instance, run the following command:

$ source /path/to/examplerc
$ openstack server add floating ip <deployment-instance-name> <ip>

For example, if instance deployment.example.com is to be assigned IP 10.20.120.150 the command would be:

$ source /path/to/examplerc
$ openstack server add floating ip deployment.example.com 10.20.120.150

Adding the RC File to the Deployment Host

Once the deployment host exists, copy the RC file created earlier to the deployment host via scp as follows

scp <rc-file-deployment-host> cloud-user@<ip>:/home/cloud-user/

Deployment Host Configuration for OKD

The following subsections describe all the steps needed to properly configure the deployment instance.

Configure ~/.ssh/config to use Deployment Host as a Jumphost

To easily connect to the OKD environment, follow the steps below.

On the OpenStack director node or local workstation with the private key, .pem:

$ exec ssh-agent bash
$ ssh-add /path/to/<keypair-name>.pem
Identity added: /path/to/<keypair-name>.pem (/path/to/<keypair-name>.pem)

Add to the ~/.ssh/config file:

Host deployment
    HostName        <deployment_fqdn_hostname OR IP address>
    User            cloud-user
    IdentityFile    /path/to/<keypair-name>.pem
    ForwardAgent     yes

ssh into the deployment host with the -A option that enables forwarding of the authentication agent connection.

Ensure the permissions are read write only for the owner of the ~/.ssh/config file:

$ chmod 600 ~/.ssh/config

$ ssh -A cloud-user@deployment

Once logged into the deployment host, verify the ssh agent forwarding is working via checking for the SSH_AUTH_SOCK

$ echo "$SSH_AUTH_SOCK"
/tmp/ssh-NDFDQD02qB/agent.1387

Subscription Manager and Enabling OKD Repositories

Within the deployment instance, register it with the Red Hat Subscription Manager. This can be accomplished by using credentials:

$ sudo subscription-manager register --username <user> --password '<password>'

Alternatively, you can use an activation key:

$ sudo subscription-manager register --org="<org_id>" --activationkey=<keyname>

Once registered, enable the following repositories as follows.

$ sudo subscription-manager repos \
    --enable="rhel-7-server-rpms" \
    --enable="rhel-7-server-extras-rpms" \
    --enable="rhel-7-server-ose-3.11-rpms" \
    --enable="rhel-7-server-ansible-2.6-rpms" \
    --enable="rhel-7-server-openstack-13-rpms" \
    --enable="rhel-7-server-openstack-13-tools-rpms"

Required Packages on the Deployment Host

The following packages are required to be installed on the deployment host.

Install the following packages:

• openshift-ansible

• python-openstackclient

• python2-heatclient

• python2-octaviaclient

• python2-shade

• python-dns

• git

• ansible

$ sudo yum -y install openshift-ansible python-openstackclient python2-heatclient python2-octaviaclient python2-shade python-dns git ansible

Configure Ansible

ansible is installed on the deployment instance to perform the registration, installation of packages, and the deployment of the OKD environment on the master and node instances.

Before running playbooks, it is important to create an ansible.cfg file to reflect the environment you wish to deploy:

$ cat ~/ansible.cfg
[defaults]
forks = 20
host_key_checking = False
remote_user = openshift
gathering = smart
fact_caching = jsonfile
fact_caching_connection = $HOME/ansible/facts
fact_caching_timeout = 600
log_path = $HOME/ansible.log
nocows = 1
callback_whitelist = profile_tasks
inventory = /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py,/home/cloud-user/inventory
[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=600s -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=false
control_path = %(directory)s/%%h-%%r
pipelining = True
timeout = 10
[persistent_connection]
connect_timeout = 30
connect_retries = 30
connect_interval = 1

The code block above can overwrite the default values in the file. Ensure to populate with the keypair that was copied to the deployment instance.

OpenShift Authentication

OKD provides the ability to use many different authentication platforms. A listing of authentication options are available at Configuring Authentication and User Agent.

Configuring the default identity provider is important as the default configuration is to Deny All.

Provisioning OKD Instances using the OpenShift Ansible Playbooks

Once the creation and configuration of the deployment host is complete, we turn to preparing the environment for the deployment of OKD using Ansible. In the following subsections, Ansible is configured and certain YAML files are modified to achieve a successful OKD on OpenStack deployment.

Preparing the Inventory for Provisioning

With the installation of the openshift-ansible package complete via our previous steps, there resides a sample-inventory directory that we will copy to our cloud-user home directory of the deployment host.

On the deployment host,

$ cp -r /usr/share/ansible/openshift-ansible/playbooks/openstack/sample-inventory/ ~/inventory

Within this inventory directory, the all.yml file contains all the different parameters that must be set in to order to achieve successful provisioning of the RHOCP instances. The OSEv3.yml file contains some references required by the all.yml file and all the available OKD cluster parameters that you can customize.

OpenShiftSDN All YAML file

The all.yml file has many options that can be modified to meet your specific needs. The information gathered in this file is for the provisioning portion of the instances required for a successful deployment of OKD. It is important to review these carefully. This document will provide a condensed version of the All YAML file and focus on the most critical parameters that need to be set for a successful deployment.

$ cat ~/inventory/group_vars/all.yml
---
openshift_openstack_clusterid: "openshift"
openshift_openstack_public_dns_domain: *"example.com"*
openshift_openstack_dns_nameservers: *["10.19.115.228"]*
openshift_openstack_public_hostname_suffix: "-public"
openshift_openstack_nsupdate_zone: "{{ openshift_openstack_public_dns_domain }}"
openshift_openstack_keypair_name: *"openshift"*
openshift_openstack_external_network_name: *"public"*
openshift_openstack_default_image_name: *"rhel75"*
## Optional (Recommended) - This removes the need for floating IPs
## on the OpenShift Cluster nodes
openshift_openstack_node_subnet_name: *<deployment-subnet-name>*
openshift_openstack_router_name: *<deployment-router-name>*
openshift_openstack_master_floating_ip: *false*
openshift_openstack_infra_floating_ip: *false*
openshift_openstack_compute_floating_ip: *false*
## End of Optional Floating IP section
openshift_openstack_num_masters: *3*
openshift_openstack_num_infra: *3*
openshift_openstack_num_cns: *0*
openshift_openstack_num_nodes: *2*
openshift_openstack_master_flavor: *"m1.master"*
openshift_openstack_default_flavor: *"m1.node"*
openshift_openstack_use_lbaas_load_balancer: *true*
openshift_openstack_docker_volume_size: "15"
# # Roll-your-own DNS
*openshift_openstack_external_nsupdate_keys:*
  public:
    *key_secret: '/alb8h0EAFWvb4i+CMA12w=='*
    *key_name: "update-key"*
    *key_algorithm: 'hmac-md5'*
    *server: '<ip-of-DNS>'*
  private:
    *key_secret: '/alb8h0EAFWvb4i+CMA12w=='*
    *key_name: "update-key"*
    *key_algorithm: 'hmac-md5'*
    *server: '<ip-of-DNS>'*
ansible_user: openshift
## cloud config
openshift_openstack_disable_root: true
openshift_openstack_user: openshift

The values above that are enclosed by asterisks (*) require modification based upon your OpenStack environment and DNS server.

In order to properly modify the DNS portion of the All YAML file, login to the DNS server and perform the following commands to capture the key name, key algorithm and key secret:

$ ssh <ip-of-DNS>
$ sudo -i
# cat /etc/named/<key-name.key>
key "update-key" {
    algorithm hmac-md5;
    secret "/alb8h0EAFWvb4i+CMA02w==";
};

KuryrSDN All YAML file

The following all.yml file enables Kuryr SDN instead of the default OpenShiftSDN. Note that the example below is a condensed version and it is important to review the default template carefully.

$ cat ~/inventory/group_vars/all.yml
---
openshift_openstack_clusterid: "openshift"
openshift_openstack_public_dns_domain: *"example.com"*
openshift_openstack_dns_nameservers: *["10.19.115.228"]*
openshift_openstack_public_hostname_suffix: "-public"
openshift_openstack_nsupdate_zone: "{{ openshift_openstack_public_dns_domain }}"
openshift_openstack_keypair_name: *"openshift"*
openshift_openstack_external_network_name: *"public"*
openshift_openstack_default_image_name: *"rhel75"*
## Optional (Recommended) - This removes the need for floating IPs
## on the OpenShift Cluster nodes
openshift_openstack_node_subnet_name: *<deployment-subnet-name>*
openshift_openstack_router_name: *<deployment-router-name>*
openshift_openstack_master_floating_ip: *false*
openshift_openstack_infra_floating_ip: *false*
openshift_openstack_compute_floating_ip: *false*
## End of Optional Floating IP section
openshift_openstack_num_masters: *3*
openshift_openstack_num_infra: *3*
openshift_openstack_num_cns: *0*
openshift_openstack_num_nodes: *2*
openshift_openstack_master_flavor: *"m1.master"*
openshift_openstack_default_flavor: *"m1.node"*
## Kuryr configuration
openshift_use_kuryr: True
openshift_use_openshift_sdn: False
use_trunk_ports: True
os_sdn_network_plugin_name: cni
openshift_node_proxy_mode: userspace
kuryr_openstack_pool_driver: nested
openshift_kuryr_precreate_subports: 5
kuryr_openstack_public_net_id: *<public_ID>*
# To disable namespace isolation, comment out the next 2 lines
openshift_kuryr_subnet_driver: namespace
openshift_kuryr_sg_driver: namespace
# If you enable namespace isolation, `default` and `openshift-monitoring` become the
# global namespaces. Global namespaces can access all namespaces. All
# namespaces can access global namespaces.
# To make other namespaces global, include them here:
kuryr_openstack_global_namespaces: default,openshift-monitoring
# If OpenStack cloud endpoints are accessible over HTTPS, provide the CA certificate
kuryr_openstack_ca: *<path-to-ca-certificate>*
openshift_master_open_ports:
- service: dns tcp
  port: 53/tcp
- service: dns udp
  port: 53/udp
openshift_node_open_ports:
- service: dns tcp
  port: 53/tcp
- service: dns udp
  port: 53/udp
# End of Kuryr configuration
openshift_openstack_use_lbaas_load_balancer: *true*
openshift_openstack_docker_volume_size: "15"
# # Roll-your-own DNS
*openshift_openstack_external_nsupdate_keys:*
  public:
    *key_secret: '/alb8h0EAFWvb4i+CMA12w=='*
    *key_name: "update-key"*
    *key_algorithm: 'hmac-md5'*
    *server: '<ip-of-DNS>'*
  private:
    *key_secret: '/alb8h0EAFWvb4i+CMA12w=='*
    *key_name: "update-key"*
    *key_algorithm: 'hmac-md5'*
    *server: '<ip-of-DNS>'*
ansible_user: openshift
## cloud config
openshift_openstack_disable_root: true
openshift_openstack_user: openshift

Configuring global namespace access

The kuryr_openstack_global_namespace parameter contains a list that defines global namespaces. By default, only the default and openshift-monitoring namespaces are included in this list.

If you are upgrading from a previous z-release of OKD 3.11, note that access to other namespaces from global namespaces is controlled by the security group *-allow_from_default.

Although the remote_group_id rule can control access to other namespaces from global namespaces, using it can cause scaling and connectivity problems. To avoid these problems, switch from using remote_group_id at *_allow_from_default to remote_ip_prefix:

1. From a command line, retrieve your networks’ subnetCIDR value:

   $ oc get kuryrnets ns-default -o yaml | grep subnetCIDR
     subnetCIDR: 10.11.13.0/24

2. Create TCP and UDP rules for this range:

   $ openstack security group rule create --remote-ip 10.11.13.0/24 --protocol tcp openshift-ansible-openshift.example.com-allow_from_default
   $ openstack security group rule create --remote-ip 10.11.13.0/24 --protocol udp openshift-ansible-openshift.example.com-allow_from_default

3. Remove the security group rule that uses remote_group_id:

   $ openstack security group show *-allow_from_default | grep remote_group_id
   $ openstack security group rule delete REMOTE_GROUP_ID

OSEv3 YAML file

The OSEv3 YAML file file specifies all the different parameters and customizations relating the installation of OpenShift.

Below is a condensed version of the file with all required variables for a successful deployment. Additional variables may be required depending on what customization is required for your specific OKD deployment.

$ cat ~/inventory/group_vars/OSEv3.yml
---
openshift_deployment_type: openshift-enterprise
openshift_release: v3.11
oreg_url: registry.access.redhat.com/openshift3/ose-${component}:${version}
openshift_examples_modify_imagestreams: true
oreg_auth_user: <oreg_auth_user>
oreg_auth_password: <oreg_auth_pw>
# The following is required if you want to deploy the Operator Lifecycle Manager (OLM)
openshift_additional_registry_credentials: [{'host':'registry.connect.redhat.com','user':'REGISTRYCONNECTUSER','password':'REGISTRYCONNECTPASSWORD','test_image':'mongodb/enterprise-operator:0.3.2'}]
openshift_master_default_subdomain: "apps.{{ (openshift_openstack_clusterid|trim == '') | ternary(openshift_openstack_public_dns_domain, openshift_openstack_clusterid + '.' + openshift_openstack_public_dns_domain) }}"
openshift_master_cluster_public_hostname: "console.{{ (openshift_openstack_clusterid|trim == '') | ternary(openshift_openstack_public_dns_domain, openshift_openstack_clusterid + '.' + openshift_openstack_public_dns_domain) }}"
#OpenStack Credentials:
openshift_cloudprovider_kind: openstack
openshift_cloudprovider_openstack_auth_url: "{{ lookup('env','OS_AUTH_URL') }}"
openshift_cloudprovider_openstack_username: "{{ lookup('env','OS_USERNAME') }}"
openshift_cloudprovider_openstack_password: "{{ lookup('env','OS_PASSWORD') }}"
openshift_cloudprovider_openstack_tenant_name: "{{ lookup('env','OS_PROJECT_NAME') }}"
openshift_cloudprovider_openstack_blockstorage_version: v2
openshift_cloudprovider_openstack_domain_name: "{{ lookup('env','OS_USER_DOMAIN_NAME') }}"
openshift_cloudprovider_openstack_conf_file: <path_to_local_openstack_configuration_file>
#Use Cinder volume for Openshift registry:
openshift_hosted_registry_storage_kind: openstack
openshift_hosted_registry_storage_access_modes: ['ReadWriteOnce']
openshift_hosted_registry_storage_openstack_filesystem: xfs
openshift_hosted_registry_storage_volume_size: 30Gi
openshift_hosted_registry_storage_openstack_volumeID: d65209f0-9061-4cd8-8827-ae6e2253a18d
openshift_hostname_check: false
ansible_become: true
#Setting SDN (defaults to ovs-networkpolicy) not part of OSEv3.yml
#For more info, on which to choose, visit:
#https://docs.openshift.com/container-platform/3.11/architecture/networking/sdn.html#overview
networkPluginName: redhat/ovs-networkpolicy
#networkPluginName: redhat/ovs-multitenant
#Configuring identity providers with Ansible
#For initial cluster installations, the Deny All identity provider is configured
#by default. It is recommended to be configured with either htpasswd
#authentication, LDAP authentication, or Allowing all authentication (not recommended)
#For more info, visit:
#https://docs.openshift.com/container-platform/3.10/install_config/configuring_authentication.html#identity-providers-ansible
#Example of Allowing All
#openshift_master_identity_providers: [{'name': 'allow_all', 'login': 'true', 'challenge': 'true', 'kind': 'AllowAllPasswordIdentityProvider'}]
#Optional Metrics (uncomment below lines for installation)
#openshift_metrics_install_metrics: true
#openshift_metrics_cassandra_storage_type: dynamic
#openshift_metrics_storage_volume_size: 25Gi
#openshift_metrics_cassandra_nodeselector: {"node-role.kubernetes.io/infra":"true"}
#openshift_metrics_hawkular_nodeselector: {"node-role.kubernetes.io/infra":"true"}
#openshift_metrics_heapster_nodeselector: {"node-role.kubernetes.io/infra":"true"}
#Optional Aggregated Logging (uncomment below lines for installation)
#openshift_logging_install_logging: true
#openshift_logging_es_pvc_dynamic: true
#openshift_logging_es_pvc_size: 30Gi
#openshift_logging_es_cluster_size: 3
#openshift_logging_es_number_of_replicas: 1
#openshift_logging_es_nodeselector: {"node-role.kubernetes.io/infra":"true"}
#openshift_logging_kibana_nodeselector: {"node-role.kubernetes.io/infra":"true"}
#openshift_logging_curator_nodeselector: {"node-role.kubernetes.io/infra":"true"}

For further details on any of the variables listed, see an example OpenShift-Ansible host inventory.

OpenStack Prerequisites Playbook

The OKD Ansible Installer provides a playbook to ensure all the provisioning steps of the OpenStack instances have been met.

Prior to running the playbook, ensure to source the RC file

$ source path/to/examplerc

Via the ansible-playbook command on the deployment host, ensure all the prerequisites are met using prerequisites.yml playbook:

$  ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/prerequisites.yml

Once the prerequisite playbook completes successfully, run the provision playbook as follows:

$ ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/provision.yml

$ watch openstack stack list
+———————————————————+—————————-+——————————+———————————+———————+
| ID                                   | Stack Name        | Stack Status       | Creation Time        | Updated Time |
+———————————————————+—————————-+——————————+———————————+———————+
| 87cb6d1c-8516-40fc-892b-49ad5cb87fac | openshift-cluster | CREATE_IN_PROGRESS | 2018-08-20T23:44:46Z | None         |
+———————————————————+—————————-+——————————+———————————+———————+

$ openstack stack failures list openshift-cluster

Stack Name Configuration

By default, the Heat stack that is created by OpenStack for the OKD cluster is named openshift-cluster. If you want to use a different name then you must set the OPENSHIFT_CLUSTER environment variable before running the playbooks:

$ export OPENSHIFT_CLUSTER=openshift.example.com

If you use a non-default stack name and run the openshift-ansible playbooks to update your deployment, you must set OPENSHIFT_CLUSTER to your stack name to avoid errors.

Registering with Subscription Manager the OKD Instances

With the nodes successfully provisioned, the next step is to ensure all the nodes are successfully registered via subscription-manager to install all the required packages for a successful OKD installation. For simplicity, a repos.yml file has been created and provided.

$ cat ~/repos.yml
---
- name: Enable the proper repositories for OpenShift installation
  hosts: OSEv3
  become: yes
  tasks:
  - name: Register with activationkey and consume subscriptions matching Red Hat Cloud Suite or Red Hat OpenShift Container Platform
    redhat_subscription:
      state: present
      activationkey: <key-name>
      org_id: <orig_id>
      pool: '^(Red Hat Cloud Suite|Red Hat OpenShift Container Platform)$'
  - name: Disable all current repositories
    rhsm_repository:
      name: '*'
      state: disabled
  - name: Enable Repositories
    rhsm_repository:
      name: "{{ item }}"
      state: enabled
    with_items:
      - rhel-7-server-rpms
      - rhel-7-server-extras-rpms
      - rhel-7-server-ansible-2.6-rpms
      - rhel-7-server-ose-3.11-rpms

With the repos.yml, run the ansible-playbook command:

$ ansible-playbook repos.yml

The above example uses Ansible’s redhat_subscription and rhsm_repository modules for all registration, disabling and enabling of repositories. This specific example takes advantage of using a Red Hat activation key. If you don’t have an activation key, ensure to visit the Ansible redhat_subscription module to modify using a username and password instead as shown in the examples: https://docs.ansible.com/ansible/2.6/modules/redhat_subscription_module.html

Installing OKD by Using an Ansible Playbook

With the OpenStack instances provisioned, the focus shifts to the installation OKD. The installation and configuration is done via a series of Ansible playbooks and roles provided by the OpenShift RPM packages. Review the OSEv3.yml file that was previous configured to ensure all the options have been properly set.

Prior to running the installer playbook, ensure all the {rhocp} prerequisites are met via:

$ ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml

Run the installer playbook to install Red Hat OpenShift Container Platform:

$ ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/install.yml

Applying Configuration Changes to Existing OKD Environment

Start or restart OKD services on all master and node hosts to apply your configuration changes, see Restarting OKD services:

# master-restart api
# master-restart controllers
# systemctl restart atomic-openshift-node

Switching from not using a cloud provider to using a cloud provider produces an error message. Adding the cloud provider tries to delete the node because the node switches from using the hostname as the **externalID** (which would have been the case when no cloud provider was being used) to using the cloud provider’s **instance-id** (which is what the cloud provider specifies). To resolve this issue:

1. Log in to the CLI as a cluster administrator.

2. Check and back up existing node labels:

   $ oc describe node <node_name> | grep -Poz '(?s)Labels.*\n.*(?=Taints)'

3. Delete the nodes:

   $ oc delete node <node_name>

4. On each node host, restart the OKD service.

   # systemctl restart origin-node

5. Add back any labels on each node that you previously had.

Configuring OpenStack Variables on an existing OpenShift Environment

To set the required OpenStack variables, modify the /etc/origin/cloudprovider/openstack.conf file with the following contents on all of your OKD hosts, both masters and nodes:

[Global]
auth-url = <OS_AUTH_URL>
username = <OS_USERNAME>
password = <password>
domain-id = <OS_USER_DOMAIN_ID>
tenant-id = <OS_TENANT_ID>
region = <OS_REGION_NAME>
[LoadBalancer]
subnet-id = <UUID of the load balancer subnet>

Consult your OpenStack administrators for values of the **OS_** variables, which are commonly used in OpenStack configuration.

Configuring Zone Labels for Dynamically Created OpenStack PVs

Administrators can configure zone labels for dynamically created OpenStack PVs. This option is useful if the OpenStack Cinder zone name does not match the compute zone names, for example, if there is only one Cinder zone and many compute zones. Administrators can create Cinder volumes dynamically and then check the labels.

To view the zone labels for the PVs:

# oc get pv --show-labels
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                 STORAGECLASS   REASON    AGE       LABELS
pvc-1faa6f93-64ac-11e8-930c-fa163e3c373c   1Gi        RWO            Delete           Bound     openshift-node/pvc1   standard                 12s       failure-domain.beta.kubernetes.io/zone=nova

The default setting is enabled. Using the oc get pv --show-labels command returns the failure-domain.beta.kubernetes.io/zone=nova label.

To disable the zone label, update the openstack.conf file by adding:

[BlockStorage]
ignore-volume-az = yes

The PVs created after restarting the master services will not have the zone label.