Installing OKD

To install a OKD cluster, you run a series of Ansible playbooks.

Running Ansible playbooks with the —tags or —check options is not supported by Red Hat.

To install OKD as a stand-alone registry, see Installing a Stand-alone Registry.

Prerequisites

Before installing OKD, prepare your cluster hosts:

  • Review the System and environment requirements.

  • If you will have a large cluster, review the Scaling and Performance Guide for suggestions for optimizing installation time.

  • Prepare your hosts. This process includes verifying system and environment requirements per component type, installing and configuring the docker service, and installing Ansible version 2.6 or later. You must install Ansible to run the installation playbooks.

  • Configure your inventory file to define your environment and OKD cluster configuration. Both your initial installation and future cluster upgrades are based on this inventory file.

  • If you are installing OKD on Red Hat Enterprise Linux, decide if you want to use the RPM or system container installation method. The system container method is required for RHEL Atomic Host systems.

Cloud installation

You can provision OKD VMs in a cloud environment. You can use Ansible playbooks to automate the definition of your cloud hosted infrastructure and application of post-provision configuration for the supported cloud providers.

OpenStack provider

You can install OKD by using the OpenStack CLI. For more information, See the reference architecture for OKD 3.6 and Red Hat OpenStack Platform 10.

As a prerequisite to using the OpenStack CLI, first provision VMs and configure the cloud infrastructure, such as networking, storage, firewall, and security groups. For information about these configuration tasks, see the cloud provider considerations and Ansible playbooks to automate them. See also Configuring for OpenStack and Configuring Your Inventory File.

Running the RPM-based installer

The RPM-based installer uses Ansible installed via RPM packages to run playbooks and configuration files available on the local host.

Do not run OpenShift Ansible playbooks under nohup. Using nohup with the playbooks causes file descriptors to be created but not closed. Therefore, the system can run out of files to open and the playbook fails.

To run the RPM-based installer:

  1. Change to the playbook directory and run the prerequisites.yml playbook. This playbook installs required software packages, if any, and modifies the container runtimes. Unless you need to configure the container runtimes, run this playbook only once, before you deploy a cluster the first time:

    1. $ cd /usr/share/ansible/openshift-ansible
    2. $ ansible-playbook [-i /path/to/inventory] \ (1)
    3.   playbooks/prerequisites.yml
    1If your inventory file is not in the /etc/ansible/hosts directory, specify -i and the path to the inventory file.

  2. Change to the playbook directory and run the deploy_cluster.yml playbook to initiate the cluster installation:

    1. $ cd /usr/share/ansible/openshift-ansible
    2. $ ansible-playbook [-i /path/to/inventory] \ (1)
    3.     playbooks/deploy_cluster.yml
    1If your inventory file is not in the /etc/ansible/hosts directory, specify -i and the path to the inventory file.

. If your installation succeeded, verify the installation. If your installation failed, retry the installation.

Running the containerized installer

The openshift/origin-ansible image is a containerized version of the OKD installer. This installer image provides the same functionality as the RPM-based installer, but it runs in a containerized environment that provides all of its dependencies rather than being installed directly on the host. The only requirement to use it is the ability to run a container.

Running the installer as a system container

The installer image can be used as a system container. System containers are stored and run outside of the traditional docker service. This enables running the installer image from one of the target hosts without concern for the install restarting docker on the host.

To use the Atomic CLI to run the installer as a run-once system container, perform the following steps as the root user:

  1. Run the prerequisites.yml playbook:

    1. # atomic install --system \
    2.     --storage=ostree \
    3.     --set INVENTORY_FILE=/path/to/inventory \ (1)
    4.     --set PLAYBOOK_FILE=/usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml \
    5.     --set OPTS="-v" \
    6.     docker.io/openshift/origin-ansible:v3.11
    1Specify the location on the local host for your inventory file.

    This command runs a set of prerequiste tasks by using the inventory file specified and the root user’s SSH configuration.

  2. Run the deploy_cluster.yml playbook:

    1. # atomic install --system \
    2.     --storage=ostree \
    3.     --set INVENTORY_FILE=/path/to/inventory \ (1)
    4.     --set PLAYBOOK_FILE=/usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml \
    5.     --set OPTS="-v" \
    6.     docker.io/openshift/origin-ansible:v3.11
    1Specify the location on the local host for your inventory file.

    This command initiates the cluster installation by using the inventory file specified and the root user’s SSH configuration. It logs the output on the terminal and also saves it in the /var/log/ansible.log file. The first time this command is run, the image is imported into OSTree storage (system containers use this rather than docker daemon storage). On subsequent runs, it reuses the stored image.

    If for any reason the installation fails, before re-running the installer, see Known Issues to check for any specific instructions or workarounds.

Running other playbooks

You can use the PLAYBOOK_FILE environment variable to specify other playbooks you want to run by using the containerized installer. The default value of the PLAYBOOK_FILE is /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml, which is the main cluster installation playbook, but you can set it to the path of another playbook inside the container.

For example, to run the pre-install checks playbook before installation, use the following command:

  1. # atomic install --system \
  2.     --storage=ostree \
  3.     --set INVENTORY_FILE=/path/to/inventory \
  4.     --set PLAYBOOK_FILE=/usr/share/ansible/openshift-ansible/playbooks/openshift-checks/pre-install.yml \ (1)
  5.     --set OPTS="-v" \ (2)
  6.     docker.io/openshift/origin-ansible:v3.11
1Set PLAYBOOK_FILE to the full path of the playbook starting at the playbooks/ directory. Playbooks are located in the same locations as with the RPM-based installer.
2Set OPTS to add command line options to ansible-playbook.

Running the installer as a container

The installer image can also run as a docker container anywhere that docker can run.

This method must not be used to run the installer on one of the hosts being configured, because the installer might restart docker on the host and disrupt the installation.

Although this method and the system container method above use the same image, they run with different entry points and contexts, so runtime parameters are not the same.

At a minimum, when running the installer as a docker container you must provide:

  • SSH key(s), so that Ansible can reach your hosts.

  • An Ansible inventory file.

  • The location of the Ansible playbook to run against that inventory.

Here is an example of how to run an install via docker, which must be run by a non-root user with access to docker:

  1. First, run the prerequisites.yml playbook:

    1. $ docker run -t -u `id -u` \ (1)
    2.     -v $HOME/.ssh/id_rsa:/opt/app-root/src/.ssh/id_rsa:Z \ (2)
    3.     -v $HOME/ansible/hosts:/tmp/inventory:Z \ (3)
    4.     -e INVENTORY_FILE=/tmp/inventory \ (3)
    5.     -e PLAYBOOK_FILE=playbooks/prerequisites.yml \ (4)
    6.     -e OPTS="-v" \ (5)
    7.     docker.io/openshift/origin-ansible:v3.11
    1-u id -u makes the container run with the same UID as the current user, which allows that user to use the SSH key inside the container. SSH private keys are expected to be readable only by their owner.
    2-v $HOME/.ssh/id_rsa:/opt/app-root/src/.ssh/id_rsa:Z mounts your SSH key, $HOME/.ssh/id_rsa, under the container user’s $HOME/.ssh directory. /opt/app-root/src is the $HOME of the user in the container. If you mount the SSH key into a different location, add an environment variable with -e ANSIBLE_PRIVATE_KEY_FILE=/the/mount/point or set ansible_ssh_private_key_file=/the/mount/point as a variable in the inventory to point Ansible to it. Note that the SSH key is mounted with the :Z flag. This flag is required so that the container can read the SSH key under its restricted SELinux context. This also means that your original SSH key file will be re-labeled to something like system_u:object_r:container_file_t:s0:c113,c247. For more details about :Z, review the docker-run(1) man page. Keep this in mind when providing these volume mount specifications because this might have unexpected consequences, For example, if you mount (and therefore re-label) your whole $HOME/.ssh directory, it will block the host’s sshd from accessing your public keys to log in. For this reason, you might want to use a separate copy of the SSH key or directory so that the original file labels remain untouched.
    3-v $HOME/ansible/hosts:/tmp/inventory:Z and -e INVENTORY_FILE=/tmp/inventory mount a static Ansible inventory file into the container as /tmp/inventory and set the corresponding environment variable to point at it. As with the SSH key, the inventory file SELinux labels might need to be relabeled by using the :Z flag to allow reading in the container, depending on the existing label. For files in a user $HOME directory, this is likely to be needed. You might prefer to copy the inventory to a dedicated location before you mount it. You can also download the inventory file from a web server if you specify the INVENTORY_URL environment variable or generate it dynamically by using the DYNAMIC_SCRIPT_URL parameter to specify an executable script that provides a dynamic inventory.
    4-e PLAYBOOK_FILE=playbooks/prerequisites.yml specifies the playbook to run as a relative path from the top level directory of openshift-ansible content. In this example, you specify the prerequisites playbook. You can also specify the full path from the RPM or the path to any other playbook file in the container.
    5-e OPTS=”-v” supplies arbitrary command line options to the ansible-playbook command that runs inside the container. In this example, specify -v to increase verbosity.

  2. Next, run the deploy_cluster.yml playbook to initiate the cluster installation:

    1. $ docker run -t -u `id -u` \
    2.     -v $HOME/.ssh/id_rsa:/opt/app-root/src/.ssh/id_rsa:Z \
    3.     -v $HOME/ansible/hosts:/tmp/inventory:Z \
    4.     -e INVENTORY_FILE=/tmp/inventory \
    5.     -e PLAYBOOK_FILE=playbooks/deploy_cluster.yml \
    6.     -e OPTS="-v" \
    7.     docker.io/openshift/origin-ansible:v3.11

Running the Installation Playbook for OpenStack

To install OKD on an existing OpenStack installation, use the OpenStack playbook. For more information about the playbook, including detailed prerequisites, see the OpenStack Provisioning readme file.

To run the playbook, run the following command:

  1. $ ansible-playbook --user openshift \
  2.   -i openshift-ansible/playbooks/openstack/inventory.py \
  3.   -i inventory \
  4.   openshift-ansible/playbooks/openstack/openshift-cluster/provision_install.yml

About the installation playbooks

The installer uses modularized playbooks so that administrators can install specific components as needed. By breaking up the roles and playbooks, there is better targeting of ad hoc administration tasks. This results in an increased level of control during installations and results in time savings.

The main installation playbook ~/openshift-ansible/playbooks/deploy_cluster.yml runs a set of individual component playbooks in a specific order, and the installer reports back at the end what phases you have gone through. If the installation fails, you are notified which phase failed along with the errors from the Ansible run.

While RHEL Atomic Host is supported for running OKD services as system container, the installation method uses Ansible, which is not available in RHEL Atomic Host. The RPM-based installer must therefore be run from a supported version of Fedora, CentOS, or RHEL. The host initiating the installation does not need to be intended for inclusion in the OKD cluster, but it can be. Alternatively, a containerized version of the installer is available as a system container, which can be run from a RHEL Atomic Host system.

Retrying the installation

If the Ansible installer fails, you can still install OKD:

  1. Review the Known Issues to check for any specific instructions or workarounds.

  2. Address the errors from your installation.

  3. Determine if you need to uninstall and reinstall or retry the installation:

    • If you did not modify the SDN configuration or generate new certificates, retry the installation.

    • If you modified the SDN configuration, generated new certificates, or the installer fails again, you must either start over with a clean operating system installation or uninstall and install again.

    • If you use virtual machines, start from a new image or uninstall and install again.

    • If you use bare metal machines, uninstall and install again.

  1. Retry the installation:

    • You can run the deploy_cluster.yml playbook again.

    • You can run the remaining individual installation playbooks.

      If you want to run only the remaining playbooks, start by running the playbook for the phase that failed and then run each of the remaining playbooks in order. Run each playbook with the following command:

      1. # ansible-playbook [-i /path/to/inventory] <playbook_file_location>

      The following table lists the playbooks in the order that they must run:

      Table 1. Individual Component Playbook Run Order
      Playbook NameFile Location

      Health Check

      ~/openshift-ansible/playbooks/openshift-checks/pre-install.yml

      Node Bootstrap

      ~/openshift-ansible/playbooks/openshift-node/bootstrap.yml

      etcd Install

      ~/openshift-ansible/playbooks/openshift-etcd/config.yml

      NFS Install

      ~/openshift-ansible/playbooks/openshift-nfs/config.yml

      Load Balancer Install

      ~/openshift-ansible/playbooks/openshift-loadbalancer/config.yml

      Master Install

      ~/openshift-ansible/playbooks/openshift-master/config.yml

      Master Additional Install

      ~/openshift-ansible/playbooks/openshift-master/additional_config.yml

      Node Join

      ~/openshift-ansible/playbooks/openshift-node/join.yml

      GlusterFS Install

      ~/openshift-ansible/playbooks/openshift-glusterfs/config.yml

      Hosted Install

      ~/openshift-ansible/playbooks/openshift-hosted/config.yml

      Monitoring Install

      ~/openshift-ansible/playbooks/openshift-monitoring/config.yml

      Web Console Install

      ~/openshift-ansible/playbooks/openshift-web-console/config.yml

      Admin Console Install

      ~/openshift-ansible/playbooks/openshift-console/config.yml

      Metrics Install

      ~/openshift-ansible/playbooks/openshift-metrics/config.yml

      metrics-server

      ~/openshift-ansible/playbooks/metrics-server/config.yml

      Logging Install

      ~/openshift-ansible/playbooks/openshift-logging/config.yml

      Availability Monitoring Install

      ~/openshift-ansible/playbooks/openshift-monitor-availability/config.yml

      Service Catalog Install

      ~/openshift-ansible/playbooks/openshift-service-catalog/config.yml

      Management Install

      ~/openshift-ansible/playbooks/openshift-management/config.yml

      Descheduler Install

      ~/openshift-ansible/playbooks/openshift-descheduler/config.yml

      Node Problem Detector Install

      ~/openshift-ansible/playbooks/openshift-node-problem-detector/config.yml

      Operator Lifecycle Manager (OLM) Install (Technology Preview)

      ~/openshift-ansible/playbooks/olm/config.yml

Verifying the Installation

After the installation completes:

  1. Verify that the master is started and nodes are registered and reporting in Ready status. On the master host, run the following command as root:

    1. # oc get nodes
    2. NAME                   STATUS    ROLES     AGE       VERSION
    3. master.example.com     Ready     master    7h        v1.9.1+a0ce1bc657
    4. node1.example.com      Ready     compute   7h        v1.9.1+a0ce1bc657
    5. node2.example.com      Ready     compute   7h        v1.9.1+a0ce1bc657

  2. To verify that the web console is installed correctly, use the master host name and the web console port number to access the web console with a web browser.

    For example, for a master host with a host name of master.openshift.com and using the default port of 8443, the web console URL is https://master.openshift.com:8443/console.

Verifying Multiple etcd Hosts

If you installed multiple etcd hosts:

  1. First, verify that the etcd package, which provides the etcdctl command, is installed:

    1. # yum install etcd

  2. On a master host, verify the etcd cluster health, substituting for the FQDNs of your etcd hosts in the following:

    1. # etcdctl -C \
    2.     https://etcd1.example.com:2379,https://etcd2.example.com:2379,https://etcd3.example.com:2379 \
    3.     --ca-file=/etc/origin/master/master.etcd-ca.crt \
    4.     --cert-file=/etc/origin/master/master.etcd-client.crt \
    5.     --key-file=/etc/origin/master/master.etcd-client.key cluster-health

  3. Also verify the member list is correct:

    1. # etcdctl -C \
    2.     https://etcd1.example.com:2379,https://etcd2.example.com:2379,https://etcd3.example.com:2379 \
    3.     --ca-file=/etc/origin/master/master.etcd-ca.crt \
    4.     --cert-file=/etc/origin/master/master.etcd-client.crt \
    5.     --key-file=/etc/origin/master/master.etcd-client.key member list

Verifying Multiple Masters Using HAProxy

If you installed multiple masters using HAProxy as a load balancer, open the following URL and check HAProxy’s status:

  1. http://<lb_hostname>:9000 (1)
1Provide the load balancer host name listed in the [lb] section of your inventory file.

You can verify your installation by consulting the HAProxy Configuration documentation.

Optionally securing builds

Running docker build is a privileged process, so the container has more access to the node than might be considered acceptable in some multi-tenant environments. If you do not trust your users, you can configure a more secure option after installation. Disable Docker builds on the cluster and require that users build images outside of the cluster. See Securing Builds by Strategy for more information about this optional process.

Known Issues

  • On failover in multiple master clusters, it is possible for the controller manager to overcorrect, which causes the system to run more pods than what was intended. However, this is a transient event and the system does correct itself over time. See https://github.com/kubernetes/kubernetes/issues/10030 for details.

  • Due to a known issue, after running the installation, if NFS volumes are provisioned for any component, the following directories might be created whether their components are being deployed to NFS volumes or not:

    • /exports/logging-es

    • /exports/logging-es-ops/

    • /exports/metrics/

    • /exports/prometheus

    • /exports/prometheus-alertbuffer/

    • /exports/prometheus-alertmanager/

      You can delete these directories after installation, as needed.

What’s Next?

Now that you have a working OKD instance, you can: