Installing a user-provisioned bare metal cluster with network customizations

• Prerequisites
• Requirements for a cluster with user-provisioned infrastructure
  • Required machines
  • Minimum resource requirements
  • Certificate signing requests management
  • Networking requirements for user-provisioned infrastructure
  • User-provisioned DNS requirements
  • Load balancing requirements for user-provisioned infrastructure
• Preparing the user-provisioned infrastructure
• Validating DNS resolution for user-provisioned infrastructure
• Generating a key pair for cluster node SSH access
• Obtaining the installation program
• Installing the OpenShift CLI by downloading the binary
• Manually creating the installation configuration file
  • Installation configuration parameters
  • Sample install-config.yaml file for bare metal
• Network configuration phases
• Specifying advanced network configuration
• Cluster Network Operator configuration
  • Cluster Network Operator configuration object
• Creating the Ignition config files
• Installing FCOS and starting the OKD bootstrap process
  • Installing FCOS by using an ISO image
  • Installing FCOS by using PXE or iPXE booting
  • Advanced FCOS installation configuration
  • Enabling multipathing with kernel arguments on FCOS
  • Updating the bootloader using bootupd
• Waiting for the bootstrap process to complete
• Logging in to the cluster by using the CLI
• Approving the certificate signing requests for your machines
• Initial Operator configuration
  • Image registry removed during installation
  • Image registry storage configuration
  • Configuring block registry storage
• Completing installation on user-provisioned infrastructure
• Next steps

In OKD 4.9, you can install a cluster on bare metal infrastructure that you provision with customized network configuration options. By customizing your network configuration, your cluster can coexist with existing IP address allocations in your environment and integrate with existing MTU and VXLAN configurations.

When you customize OKD networking, you must set most of the network configuration parameters during installation. You can modify only kubeProxy network configuration parameters in a running cluster.

Prerequisites

• You reviewed details about the OKD installation and update processes.

• You read the documentation on selecting a cluster installation method and preparing it for users.

• If you use a firewall and plan to use the Telemetry service, you configured the firewall to allow the sites that your cluster requires access to.

Additional resources

• See Installing a user-provisioned bare metal cluster on a restricted network for more information about performing a restricted network installation on bare metal infrastructure that you provision.

Requirements for a cluster with user-provisioned infrastructure

For a cluster that contains user-provisioned infrastructure, you must deploy all of the required machines.

This section describes the requirements for deploying OKD on user-provisioned infrastructure.

Required machines

The smallest OKD clusters require the following hosts:

The bootstrap and control plane machines must use Fedora CoreOS (FCOS) as the operating system. However, the compute machines can choose between Fedora CoreOS (FCOS), Fedora 7.9, or Fedora 8.4.

See Red Hat Enterprise Linux technology capabilities and limits.

Minimum resource requirements

Each cluster machine must meet the following minimum requirements:

1. One CPU is equivalent to one physical core when simultaneous multithreading (SMT), or hyperthreading, is not enabled. When enabled, use the following formula to calculate the corresponding ratio: (threads per core × cores) × sockets = CPUs.

2. OKD and Kubernetes are sensitive to disk performance, and faster storage is recommended, particularly for etcd on the control plane nodes which require a 10 ms p99 fsync duration. Note that on many cloud platforms, storage size and IOPS scale together, so you might need to over-allocate storage volume to obtain sufficient performance.

3. As with all user-provisioned installations, if you choose to use Fedora compute machines in your cluster, you take responsibility for all operating system life cycle management and maintenance, including performing system updates, applying patches, and completing all other required tasks. Use of Fedora 7 compute machines is deprecated and planned for removal in a future release of OKD 4.

Certificate signing requests management

Because your cluster has limited access to automatic machine management when you use infrastructure that you provision, you must provide a mechanism for approving cluster certificate signing requests (CSRs) after installation. The kube-controller-manager only approves the kubelet client CSRs. The machine-approver cannot guarantee the validity of a serving certificate that is requested by using kubelet credentials because it cannot confirm that the correct machine issued the request. You must determine and implement a method of verifying the validity of the kubelet serving certificate requests and approving them.

Additional resources

• See Configuring a three-node cluster for details about deploying three-node clusters in bare metal environments.

• See Approving the certificate signing requests for your machines for more information about approving cluster certificate signing requests after installation.

Networking requirements for user-provisioned infrastructure

All the Fedora CoreOS (FCOS) machines require networking to be configured in initramfs during boot to fetch their Ignition config files.

During the initial boot, the machines require an IP address configuration that is set either through a DHCP server or statically by providing the required boot options. After a network connection is established, the machines download their Ignition config files from an HTTP or HTTPS server. The Ignition config files are then used to set the exact state of each machine. The Machine Config Operator completes more changes to the machines, such as the application of new certificates or keys, after installation.

It is recommended to use a DHCP server for long-term management of the cluster machines. Ensure that the DHCP server is configured to provide persistent IP addresses, DNS server information, and hostnames to the cluster machines.

The Kubernetes API server must be able to resolve the node names of the cluster machines. If the API servers and worker nodes are in different zones, you can configure a default DNS search zone to allow the API server to resolve the node names. Another supported approach is to always refer to hosts by their fully-qualified domain names in both the node objects and all DNS requests.

Setting the cluster node hostnames through DHCP

On Fedora CoreOS (FCOS) machines, the hostname is set through NetworkManager. By default, the machines obtain their hostname through DHCP. If the hostname is not provided by DHCP, it is obtained through a reverse DNS lookup. Reverse DNS lookup occurs after the network has been initialized on a node and can take time to resolve. Other system services can start prior to this and detect the hostname as localhost or similar. You can avoid this by using DHCP to provide the hostname for each cluster node.

Additionally, setting the hostnames through DHCP can bypass any manual DNS record name configuration errors in environments that have a DNS split-horizon implementation.

Network connectivity requirements

You must configure the network connectivity between machines to allow OKD cluster components to communicate. Each machine must be able to resolve the hostnames of all other machines in the cluster.

This section provides details about the ports that are required.

NTP configuration for user-provisioned infrastructure

OKD clusters are configured to use a public Network Time Protocol (NTP) server by default. If you want to use a local enterprise NTP server, or if your cluster is being deployed in a disconnected network, you can configure the cluster to use a specific time server. For more information, see the documentation for Configuring chrony time service.

If a DHCP server provides NTP server information, the chrony time service on the Fedora CoreOS (FCOS) machines read the information and can sync the clock with the NTP servers.

Additional resources

• Configuring chrony time service

User-provisioned DNS requirements

In OKD deployments, DNS name resolution is required for the following components:

• The Kubernetes API

• The OKD application wildcard

• The bootstrap, control plane, and compute machines

Reverse DNS resolution is also required for the Kubernetes API, the bootstrap machine, the control plane machines, and the compute machines.

DNS A/AAAA or CNAME records are used for name resolution and PTR records are used for reverse name resolution. The reverse records are important because Fedora CoreOS (FCOS) uses the reverse records to set the hostnames for all the nodes, unless the hostnames are provided by DHCP. Additionally, the reverse records are used to generate the certificate signing requests (CSR) that OKD needs to operate.

The following DNS records are required for a user-provisioned OKD cluster and they must be in place before installation. In each record, <cluster_name> is the cluster name and <base_domain> is the base domain that you specify in the install-config.yaml file. A complete DNS record takes the form: <component>.<cluster_name>.<base_domain>..

Example DNS configuration for user-provisioned clusters

This section provides A and PTR record configuration samples that meet the DNS requirements for deploying OKD on user-provisioned infrastructure. The samples are not meant to provide advice for choosing one DNS solution over another.

In the examples, the cluster name is ocp4 and the base domain is example.com.

Example DNS A record configuration for a user-provisioned cluster

The following example is a BIND zone file that shows sample A records for name resolution in a user-provisioned cluster.

Sample DNS zone database

$TTL 1W
@    IN    SOA    ns1.example.com.    root (
            2019070700    ; serial
            3H        ; refresh (3 hours)
            30M        ; retry (30 minutes)
            2W        ; expiry (2 weeks)
            1W )        ; minimum (1 week)
    IN    NS    ns1.example.com.
    IN    MX 10    smtp.example.com.
;
;
ns1.example.com.        IN    A    192.168.1.5
smtp.example.com.        IN    A    192.168.1.5
;
helper.example.com.        IN    A    192.168.1.5
helper.ocp4.example.com.    IN    A    192.168.1.5
;
api.ocp4.example.com.        IN    A    192.168.1.5 (1)
api-int.ocp4.example.com.    IN    A    192.168.1.5 (2)
;
*.apps.ocp4.example.com.    IN    A    192.168.1.5 (3)
;
bootstrap.ocp4.example.com.    IN    A    192.168.1.96 (4)
;
master0.ocp4.example.com.    IN    A    192.168.1.97 (5)
master1.ocp4.example.com.    IN    A    192.168.1.98 (5)
master2.ocp4.example.com.    IN    A    192.168.1.99 (5)
;
worker0.ocp4.example.com.    IN    A    192.168.1.11 (6)
worker1.ocp4.example.com.    IN    A    192.168.1.7 (6)
;
;EOF

Example DNS PTR record configuration for a user-provisioned cluster

The following example BIND zone file shows sample PTR records for reverse name resolution in a user-provisioned cluster.

Sample DNS zone database for reverse records

$TTL 1W
@    IN    SOA    ns1.example.com.    root (
            2019070700    ; serial
            3H        ; refresh (3 hours)
            30M        ; retry (30 minutes)
            2W        ; expiry (2 weeks)
            1W )        ; minimum (1 week)
    IN    NS    ns1.example.com.
;
5.1.168.192.in-addr.arpa.    IN    PTR    api.ocp4.example.com. (1)
5.1.168.192.in-addr.arpa.    IN    PTR    api-int.ocp4.example.com. (2)
;
96.1.168.192.in-addr.arpa.    IN    PTR    bootstrap.ocp4.example.com. (3)
;
97.1.168.192.in-addr.arpa.    IN    PTR    master0.ocp4.example.com. (4)
98.1.168.192.in-addr.arpa.    IN    PTR    master1.ocp4.example.com. (4)
99.1.168.192.in-addr.arpa.    IN    PTR    master2.ocp4.example.com. (4)
;
11.1.168.192.in-addr.arpa.    IN    PTR    worker0.ocp4.example.com. (5)
7.1.168.192.in-addr.arpa.    IN    PTR    worker1.ocp4.example.com. (5)
;
;EOF

• Validating DNS resolution for user-provisioned infrastructure

Load balancing requirements for user-provisioned infrastructure

Before you install OKD, you must provision the API and application ingress load balancing infrastructure. In production scenarios, you can deploy the API and application ingress load balancers separately so that you can scale the load balancer infrastructure for each in isolation.

The load balancing infrastructure must meet the following requirements:

1. API load balancer: Provides a common endpoint for users, both human and machine, to interact with and configure the platform. Configure the following conditions:

   • Layer 4 load balancing only. This can be referred to as Raw TCP, SSL Passthrough, or SSL Bridge mode. If you use SSL Bridge mode, you must enable Server Name Indication (SNI) for the API routes.

   • A stateless load balancing algorithm. The options vary based on the load balancer implementation.

   Session persistence is not required for the API load balancer to function properly.

   Configure the following ports on both the front and back of the load balancers:

   Table 6. API load balancer

   Port	Back-end machines (pool members)	Internal	External	Description
   6443	Bootstrap and control plane. You remove the bootstrap machine from the load balancer after the bootstrap machine initializes the cluster control plane. You must configure the /readyz endpoint for the API server health check probe.	X	X	Kubernetes API server
   22623	Bootstrap and control plane. You remove the bootstrap machine from the load balancer after the bootstrap machine initializes the cluster control plane.	X		Machine config server

   The load balancer must be configured to take a maximum of 30 seconds from the time the API server turns off the /readyz endpoint to the removal of the API server instance from the pool. Within the time frame after /readyz returns an error or becomes healthy, the endpoint must have been removed or added. Probing every 5 or 10 seconds, with two successful requests to become healthy and three to become unhealthy, are well-tested values.

2. Application ingress load balancer: Provides an ingress point for application traffic flowing in from outside the cluster. Configure the following conditions:

   • Layer 4 load balancing only. This can be referred to as Raw TCP, SSL Passthrough, or SSL Bridge mode. If you use SSL Bridge mode, you must enable Server Name Indication (SNI) for the ingress routes.

   • A connection-based or session-based persistence is recommended, based on the options available and types of applications that will be hosted on the platform.

   If the true IP address of the client can be seen by the application ingress load balancer, enabling source IP-based session persistence can improve performance for applications that use end-to-end TLS encryption.

   Configure the following ports on both the front and back of the load balancers:

   Table 7. Application ingress load balancer

   Port	Back-end machines (pool members)	Internal	External	Description
   443	The machines that run the Ingress Controller pods, compute, or worker, by default.	X	X	HTTPS traffic
   80	The machines that run the Ingress Controller pods, compute, or worker, by default.	X	X	HTTP traffic

Example load balancer configuration for user-provisioned clusters

This section provides an example API and application ingress load balancer configuration that meets the load balancing requirements for user-provisioned clusters. The sample is an /etc/haproxy/haproxy.cfg configuration for an HAProxy load balancer. The example is not meant to provide advice for choosing one load balancing solution over another.

Sample API and application ingress load balancer configuration

global
  log         127.0.0.1 local2
  pidfile     /var/run/haproxy.pid
  maxconn     4000
  daemon
defaults
  mode                    http
  log                     global
  option                  dontlognull
  option http-server-close
  option                  redispatch
  retries                 3
  timeout http-request    10s
  timeout queue           1m
  timeout connect         10s
  timeout client          1m
  timeout server          1m
  timeout http-keep-alive 10s
  timeout check           10s
  maxconn                 3000
frontend stats
  bind *:1936
  mode            http
  log             global
  maxconn 10
  stats enable
  stats hide-version
  stats refresh 30s
  stats show-node
  stats show-desc Stats for ocp4 cluster (1)
  stats auth admin:ocp4
  stats uri /stats
listen api-server-6443 (2)
  bind *:6443
  mode tcp
  server bootstrap bootstrap.ocp4.example.com:6443 check inter 1s backup (3)
  server master0 master0.ocp4.example.com:6443 check inter 1s
  server master1 master1.ocp4.example.com:6443 check inter 1s
  server master2 master2.ocp4.example.com:6443 check inter 1s
listen machine-config-server-22623 (4)
  bind *:22623
  mode tcp
  server bootstrap bootstrap.ocp4.example.com:22623 check inter 1s backup (3)
  server master0 master0.ocp4.example.com:22623 check inter 1s
  server master1 master1.ocp4.example.com:22623 check inter 1s
  server master2 master2.ocp4.example.com:22623 check inter 1s
listen ingress-router-443 (5)
  bind *:443
  mode tcp
  balance source
  server worker0 worker0.ocp4.example.com:443 check inter 1s
  server worker1 worker1.ocp4.example.com:443 check inter 1s
listen ingress-router-80 (6)
  bind *:80
  mode tcp
  balance source
  server worker0 worker0.ocp4.example.com:80 check inter 1s
  server worker1 worker1.ocp4.example.com:80 check inter 1s

Preparing the user-provisioned infrastructure

Before you install OKD on user-provisioned infrastructure, you must prepare the underlying infrastructure.

This section provides details about the high-level steps required to set up your cluster infrastructure in preparation for an OKD installation. This includes configuring IP networking and network connectivity for your cluster nodes, enabling the required ports through your firewall, and setting up the required DNS and load balancing infrastructure.

After preparation, your cluster infrastructure must meet the requirements outlined in the Requirements for a cluster with user-provisioned infrastructure section.

Prerequisites

• You have reviewed the OKD 4.x Tested Integrations page.

• You have reviewed the infrastructure requirements detailed in the Requirements for a cluster with user-provisioned infrastructure section.

Procedure

1. If you are using DHCP to provide the IP networking configuration to your cluster nodes, configure your DHCP service.

   1. Add persistent IP addresses for the nodes to your DHCP server configuration. In your configuration, match the MAC address of the relevant network interface to the intended IP address for each node.

   2. When you use DHCP to configure IP addressing for the cluster machines, the machines also obtain the DNS server information through DHCP. Define the persistent DNS server address that is used by the cluster nodes through your DHCP server configuration.

      If you are not using a DHCP service, you must provide the IP networking configuration and the address of the DNS server to the nodes at FCOS install time. These can be passed as boot arguments if you are installing from an ISO image. See the Installing FCOS and starting the OKD bootstrap process section for more information about static IP provisioning and advanced networking options.

   3. Define the hostnames of your cluster nodes in your DHCP server configuration. See the Setting the cluster node hostnames through DHCP section for details about hostname considerations.

      If you are not using a DHCP service, the cluster nodes obtain their hostname through a reverse DNS lookup.

2. Ensure that your network infrastructure provides the required network connectivity between the cluster components. See the Networking requirements for user-provisioned infrastructure section for details about the requirements.

3. Configure your firewall to enable the ports required for the OKD cluster components to communicate. See Networking requirements for user-provisioned infrastructure section for details about the ports that are required.

4. Setup the required DNS infrastructure for your cluster.

   1. Configure DNS name resolution for the Kubernetes API, the application wildcard, the bootstrap machine, the control plane machines, and the compute machines.

   2. Configure reverse DNS resolution for the Kubernetes API, the bootstrap machine, the control plane machines, and the compute machines.

      See the User-provisioned DNS requirements section for more information about the OKD DNS requirements.

5. Validate your DNS configuration.

   1. From your installation node, run DNS lookups against the record names of the Kubernetes API, the wildcard routes, and the cluster nodes. Validate that the IP addresses in the responses correspond to the correct components.

   2. From your installation node, run reverse DNS lookups against the IP addresses of the load balancer and the cluster nodes. Validate that the record names in the responses correspond to the correct components.

      See the Validating DNS resolution for user-provisioned infrastructure section for detailed DNS validation steps.

6. Provision the required API and application ingress load balancing infrastructure. See the Load balancing requirements for user-provisioned infrastructure section for more information about the requirements.

Additional resources

• Requirements for a cluster with user-provisioned infrastructure

• Installing FCOS and starting the OKD bootstrap process

• Setting the cluster node hostnames through DHCP

• Advanced RHCOS installation configuration

• Networking requirements for user-provisioned infrastructure

• User-provisioned DNS requirements

• Validating DNS resolution for user-provisioned infrastructure

• Load balancing requirements for user-provisioned infrastructure

Validating DNS resolution for user-provisioned infrastructure

You can validate your DNS configuration before installing OKD on user-provisioned infrastructure.

Prerequisites

• You have configured the required DNS records for your user-provisioned infrastructure.

Procedure

1. From your installation node, run DNS lookups against the record names of the Kubernetes API, the wildcard routes, and the cluster nodes. Validate that the IP addresses contained in the responses correspond to the correct components.

   1. Perform a lookup against the Kubernetes API record name. Check that the result points to the IP address of the API load balancer:

      $ dig +noall +answer @<nameserver_ip> api.<cluster_name>.<base_domain> (1)

      1	Replace <nameserver_ip> with the IP address of the nameserver, <cluster_name> with your cluster name, and <base_domain> with your base domain name.

      Example output

      api.ocp4.example.com.        0    IN    A    192.168.1.5

   2. Perform a lookup against the Kubernetes internal API record name. Check that the result points to the IP address of the API load balancer:

      $ dig +noall +answer @<nameserver_ip> api-int.<cluster_name>.<base_domain>

      Example output

      api-int.ocp4.example.com.        0    IN    A    192.168.1.5

   3. Test an example *.apps.<cluster_name>.<base_domain> DNS wildcard lookup. All of the application wildcard lookups must resolve to the IP address of the application ingress load balancer:

      $ dig +noall +answer @<nameserver_ip> random.apps.<cluster_name>.<base_domain>

      Example output

      random.apps.ocp4.example.com.        0    IN    A    192.168.1.5

      In the example outputs, the same load balancer is used for the Kubernetes API and application ingress traffic. In production scenarios, you can deploy the API and application ingress load balancers separately so that you can scale the load balancer infrastructure for each in isolation.

      You can replace random with another wildcard value. For example, you can query the route to the OKD console:

      $ dig +noall +answer @<nameserver_ip> console-openshift-console.apps.<cluster_name>.<base_domain>

      Example output

      console-openshift-console.apps.ocp4.example.com. 0 IN    A 192.168.1.5

   4. Run a lookup against the bootstrap DNS record name. Check that the result points to the IP address of the bootstrap node:

      $ dig +noall +answer @<nameserver_ip> bootstrap.<cluster_name>.<base_domain>

      Example output

      bootstrap.ocp4.example.com.        0    IN    A    192.168.1.96

   5. Use this method to perform lookups against the DNS record names for the control plane and compute nodes. Check that the results correspond to the IP addresses of each node.

2. From your installation node, run reverse DNS lookups against the IP addresses of the load balancer and the cluster nodes. Validate that the record names contained in the responses correspond to the correct components.

   1. Perform a reverse lookup against the IP address of the API load balancer. Check that the response includes the record names for the Kubernetes API and the Kubernetes internal API:

      $ dig +noall +answer @<nameserver_ip> -x 192.168.1.5

      Example output

      5.1.168.192.in-addr.arpa. 0    IN    PTR    api-int.ocp4.example.com. (1)
      5.1.168.192.in-addr.arpa. 0    IN    PTR    api.ocp4.example.com. (2)

      1	Provides the record name for the Kubernetes internal API.
      2	Provides the record name for the Kubernetes API.

      A PTR record is not required for the OKD application wildcard. No validation step is needed for reverse DNS resolution against the IP address of the application ingress load balancer.

   2. Perform a reverse lookup against the IP address of the bootstrap node. Check that the result points to the DNS record name of the bootstrap node:

      $ dig +noall +answer @<nameserver_ip> -x 192.168.1.96

      Example output

      96.1.168.192.in-addr.arpa. 0    IN    PTR    bootstrap.ocp4.example.com.

   3. Use this method to perform reverse lookups against the IP addresses for the control plane and compute nodes. Check that the results correspond to the DNS record names of each node.

Additional resources

• User-provisioned DNS requirements

• Load balancing requirements for user-provisioned infrastructure

Generating a key pair for cluster node SSH access

During an OKD installation, you can provide an SSH public key to the installation program. The key is passed to the Fedora CoreOS (FCOS) nodes through their Ignition config files and is used to authenticate SSH access to the nodes. The key is added to the ~/.ssh/authorized_keys list for the core user on each node, which enables password-less authentication.

After the key is passed to the nodes, you can use the key pair to SSH in to the FCOS nodes as the user core. To access the nodes through SSH, the private key identity must be managed by SSH for your local user.

If you want to SSH in to your cluster nodes to perform installation debugging or disaster recovery, you must provide the SSH public key during the installation process. The ./openshift-install gather command also requires the SSH public key to be in place on the cluster nodes.

Procedure

1. If you do not have an existing SSH key pair on your local machine to use for authentication onto your cluster nodes, create one. For example, on a computer that uses a Linux operating system, run the following command:

   $ ssh-keygen -t ed25519 -N '' -f <path>/<file_name> (1)

   1	Specify the path and file name, such as ~/.ssh/id_rsa, of the new SSH key. If you have an existing key pair, ensure your public key is in the your ~/.ssh directory.

   If you plan to install an OKD cluster that uses FIPS Validated / Modules in Process cryptographic libraries on the x86_64 architecture, do not create a key that uses the ed25519 algorithm. Instead, create a key that uses the rsa or ecdsa algorithm.

2. View the public SSH key:

   $ cat <path>/<file_name>.pub

   For example, run the following to view the ~/.ssh/id_rsa.pub public key:

   $ cat ~/.ssh/id_rsa.pub

3. Add the SSH private key identity to the SSH agent for your local user, if it has not already been added. SSH agent management of the key is required for password-less SSH authentication onto your cluster nodes, or if you want to use the ./openshift-install gather command.

   On some distributions, default SSH private key identities such as ~/.ssh/id_rsa and ~/.ssh/id_dsa are managed automatically.

   1. If the ssh-agent process is not already running for your local user, start it as a background task:

      $ eval "$(ssh-agent -s)"

      Example output

      Agent pid 31874

      If your cluster is in FIPS mode, only use FIPS-compliant algorithms to generate the SSH key. The key must be either RSA or ECDSA.

4. Add your SSH private key to the ssh-agent:

   $ ssh-add <path>/<file_name> (1)

   1	Specify the path and file name for your SSH private key, such as ~/.ssh/id_rsa

   Example output

   Identity added: /home/<you>/<path>/<file_name> (<computer_name>)

Next steps

• When you install OKD, provide the SSH public key to the installation program.

Additional resources

• Verifying node health

Obtaining the installation program

Before you install OKD, download the installation file on a local computer.

Prerequisites

• You have a computer that runs Linux or macOS, with 500 MB of local disk space

Procedure

1. Download installer from https://github.com/openshift/okd/releases

   The installation program creates several files on the computer that you use to install your cluster. You must keep the installation program and the files that the installation program creates after you finish installing the cluster. Both files are required to delete the cluster.

   Deleting the files created by the installation program does not remove your cluster, even if the cluster failed during installation. To remove your cluster, complete the OKD uninstallation procedures for your specific cloud provider.

2. Extract the installation program. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar -xvf openshift-install-linux.tar.gz

3. From the Pull Secret page on the Red Hat OpenShift Cluster Manager site, download your installation pull secret. This pull secret allows you to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OKD components.

   Using a pull secret from the Red Hat OpenShift Cluster Manager site is not required. You can use a pull secret for another private registry. Or, if you do not need the cluster to pull images from a private registry, you can use {"auths":{"fake":{"auth":"aWQ6cGFzcwo="}}} as the pull secret when prompted during the installation.

   If you do not use the pull secret from the Red Hat OpenShift Cluster Manager site:

   • Red Hat Operators are not available.

   • The Telemetry and Insights operators do not send data to Red Hat.

   • Content from the Red Hat Container Catalog registry, such as image streams and Operators, are not available.

Installing the OpenShift CLI by downloading the binary

You can install the OpenShift CLI (oc) to interact with OKD from a command-line interface. You can install oc on Linux, Windows, or macOS.

Installing the OpenShift CLI on Linux

You can install the OpenShift CLI (oc) binary on Linux by using the following procedure.

Procedure

1. Navigate to https://mirror.openshift.com/pub/openshift-v4/clients/oc/latest/ and choose the folder for your operating system and architecture.

2. Download oc.tar.gz.

3. Unpack the archive:

   $ tar xvzf <file>

4. Place the oc binary in a directory that is on your PATH.

   To check your PATH, execute the following command:

   $ echo $PATH

After you install the OpenShift CLI, it is available using the oc command:

$ oc <command>

Installing the OpenShift CLI on Windows

You can install the OpenShift CLI (oc) binary on Windows by using the following procedure.

Procedure

1. Navigate to https://mirror.openshift.com/pub/openshift-v4/clients/oc/latest/ and choose the folder for your operating system and architecture.

2. Download oc.zip.

3. Unzip the archive with a ZIP program.

4. Move the oc binary to a directory that is on your PATH.

   To check your PATH, open the command prompt and execute the following command:

   C:\> path

After you install the OpenShift CLI, it is available using the oc command:

C:\> oc <command>

Installing the OpenShift CLI on macOS

You can install the OpenShift CLI (oc) binary on macOS by using the following procedure.

Procedure

1. Navigate to https://mirror.openshift.com/pub/openshift-v4/clients/oc/latest/ and choose the folder for your operating system and architecture.

2. Download oc.tar.gz.

3. Unpack and unzip the archive.

4. Move the oc binary to a directory on your PATH.

   To check your PATH, open a terminal and execute the following command:

   $ echo $PATH

After you install the OpenShift CLI, it is available using the oc command:

$ oc <command>

Manually creating the installation configuration file

For user-provisioned installations of OKD, you manually generate your installation configuration file.

Prerequisites

• You have an SSH public key on your local machine to provide to the installation program. The key will be used for SSH authentication onto your cluster nodes for debugging and disaster recovery.

• You have obtained the OKD installation program and the pull secret for your cluster.

Procedure

1. Create an installation directory to store your required installation assets in:

   $ mkdir <installation_directory>

   You must create a directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so you must not reuse an installation directory. If you want to reuse individual files from another cluster installation, you can copy them into your directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OKD version.

2. Customize the sample install-config.yaml file template that is provided and save it in the <installation_directory>.

   You must name this configuration file install-config.yaml.

   For some platform types, you can alternatively run ./openshift-install create install-config —dir <installation_directory> to generate an install-config.yaml file. You can provide details about your cluster configuration at the prompts.

3. Back up the install-config.yaml file so that you can use it to install multiple clusters.

   The install-config.yaml file is consumed during the next step of the installation process. You must back it up now.

Installation configuration parameters

Before you deploy an OKD cluster, you provide parameter values to describe your account on the cloud platform that hosts your cluster and optionally customize your cluster’s platform. When you create the install-config.yaml installation configuration file, you provide values for the required parameters through the command line. If you customize your cluster, you can modify the install-config.yaml file to provide more details about the platform.

Required configuration parameters

Required installation configuration parameters are described in the following table:

Network configuration parameters

You can customize your installation configuration based on the requirements of your existing network infrastructure. For example, you can expand the IP address block for the cluster network or provide different IP address blocks than the defaults.

Only IPv4 addresses are supported.

networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23

networking:
  serviceNetwork:
   - 172.30.0.0/16

networking:
  machineNetwork:
  - cidr: 10.0.0.0/16

Optional configuration parameters

Optional installation configuration parameters are described in the following table:

sshKey:
  <key1>
  <key2>
  <key3>

Sample install-config.yaml file for bare metal

You can customize the install-config.yaml file to specify more details about your OKD cluster’s platform or modify the values of the required parameters.

apiVersion: v1
baseDomain: example.com (1)
compute: (2)
- hyperthreading: Enabled (3)
  name: worker
  replicas: 0 (4)
controlPlane: (2)
  hyperthreading: Enabled (3)
  name: master
  replicas: 3 (5)
metadata:
  name: test (6)
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14 (7)
    hostPrefix: 23 (8)
  networkType: OVNKubernetes
  serviceNetwork: (9)
  - 172.30.0.0/16
platform:
  none: {} (10)
pullSecret: '{"auths": ...}' (11)
sshKey: 'ssh-ed25519 AAAA...' (12)

Additional resources

• See Load balancing requirements for user-provisioned infrastructure for more information on the API and application ingress load balancing requirements.

Network configuration phases

There are two phases prior to OKD installation where you can customize the network configuration.

Phase 1

You can customize the following network-related fields in the install-config.yaml file before you create the manifest files:

• networking.networkType

• networking.clusterNetwork

• networking.serviceNetwork

• networking.machineNetwork

  For more information on these fields, refer to Installation configuration parameters.

  Set the networking.machineNetwork to match the CIDR that the preferred NIC resides in.

Phase 2

After creating the manifest files by running openshift-install create manifests, you can define a customized Cluster Network Operator manifest with only the fields you want to modify. You can use the manifest to specify advanced network configuration.

You cannot override the values specified in phase 1 in the install-config.yaml file during phase 2. However, you can further customize the cluster network provider during phase 2.

Specifying advanced network configuration

You can use advanced network configuration for your cluster network provider to integrate your cluster into your existing network environment. You can specify advanced network configuration only before you install the cluster.

Prerequisites

• You have created the install-config.yaml file and completed any modifications to it.

Procedure

1. Change to the directory that contains the installation program and create the manifests:

   $ ./openshift-install create manifests --dir <installation_directory> (1)

   1	<installation_directory> specifies the name of the directory that contains the install-config.yaml file for your cluster.

2. Create a stub manifest file for the advanced network configuration that is named cluster-network-03-config.yml in the <installation_directory>/manifests/ directory:

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:

3. Specify the advanced network configuration for your cluster in the cluster-network-03-config.yml file, such as in the following examples:

   Specify a different VXLAN port for the OpenShift SDN network provider

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     defaultNetwork:
       openshiftSDNConfig:
         vxlanPort: 4800

   Enable IPsec for the OVN-Kubernetes network provider

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     defaultNetwork:
       ovnKubernetesConfig:
         ipsecConfig: {}

4. Optional: Back up the manifests/cluster-network-03-config.yml file. The installation program consumes the manifests/ directory when you create the Ignition config files.

Cluster Network Operator configuration

The configuration for the cluster network is specified as part of the Cluster Network Operator (CNO) configuration and stored in a custom resource (CR) object that is named cluster. The CR specifies the fields for the Network API in the operator.openshift.io API group.

The CNO configuration inherits the following fields during cluster installation from the Network API in the Network.config.openshift.io API group and these fields cannot be changed:

clusterNetwork

IP address pools from which pod IP addresses are allocated.

serviceNetwork

IP address pool for services.

defaultNetwork.type

Cluster network provider, such as OpenShift SDN or OVN-Kubernetes.

You can specify the cluster network provider configuration for your cluster by setting the fields for the defaultNetwork object in the CNO object named cluster.

Cluster Network Operator configuration object

The fields for the Cluster Network Operator (CNO) are described in the following table:

spec:
  clusterNetwork:
  - cidr: 10.128.0.0/19
    hostPrefix: 23
  - cidr: 10.128.32.0/19
    hostPrefix: 23

spec:
  serviceNetwork:
  - 172.30.0.0/14

defaultNetwork object configuration

The values for the defaultNetwork object are defined in the following table:

Configuration for the OpenShift SDN CNI cluster network provider

The following table describes the configuration fields for the OpenShift SDN Container Network Interface (CNI) cluster network provider.

Example OpenShift SDN configuration

defaultNetwork:
  type: OpenShiftSDN
  openshiftSDNConfig:
    mode: NetworkPolicy
    mtu: 1450
    vxlanPort: 4789

Configuration for the OVN-Kubernetes CNI cluster network provider

The following table describes the configuration fields for the OVN-Kubernetes CNI cluster network provider.

Example OVN-Kubernetes configuration

defaultNetwork:
  type: OVNKubernetes
  ovnKubernetesConfig:
    mtu: 1400
    genevePort: 6081
    ipsecConfig: {}

kubeProxyConfig object configuration

The values for the kubeProxyConfig object are defined in the following table:

kubeProxyConfig:
  proxyArguments:
    iptables-min-sync-period:
    - 0s

Creating the Ignition config files

Because you must manually start the cluster machines, you must generate the Ignition config files that the cluster needs to make its machines.

Prerequisites

• Obtain the OKD installation program and the pull secret for your cluster.

Procedure

• Obtain the Ignition config files:

  $ ./openshift-install create ignition-configs --dir <installation_directory> (1)

  1	For <installation_directory>, specify the directory name to store the files that the installation program creates.

  If you created an install-config.yaml file, specify the directory that contains it. Otherwise, specify an empty directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so you must not reuse an installation directory. If you want to reuse individual files from another cluster installation, you can copy them into your directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OKD version.

  The following files are generated in the directory:

  .
  ├── auth
  │   ├── kubeadmin-password
  │   └── kubeconfig
  ├── bootstrap.ign
  ├── master.ign
  ├── metadata.json
  └── worker.ign

Installing FCOS and starting the OKD bootstrap process

To install OKD on bare metal infrastructure that you provision, you must install Fedora CoreOS (FCOS) on the machines. When you install FCOS, you must provide the Ignition config file that was generated by the OKD installation program for the type of machine you are installing. If you have configured suitable networking, DNS, and load balancing infrastructure, the OKD bootstrap process begins automatically after the FCOS machines have rebooted.

To install FCOS on the machines, follow either the steps to use an ISO image or network PXE booting.

You can configure FCOS during ISO and PXE installations by using the following methods:

• Kernel arguments: You can use kernel arguments to provide installation-specific information. For example, you can specify the locations of the FCOS installation files that you uploaded to your HTTP server and the location of the Ignition config file for the type of node you are installing. For a PXE installation, you can use the APPEND parameter to pass the arguments to the kernel of the live installer. For an ISO installation, you can interrupt the live installation boot process to add the kernel arguments. In both installation cases, you can use special coreos.inst.* arguments to direct the installer, as well as standard boot arguments for turning standard kernel services on or off.

• Ignition configs: OKD Ignition config files (*.ign) are specific to the type of node you are installing. You pass the location of a bootstrap, control plane, or compute node Ignition config file during the FCOS installation so that it takes effect on first boot. In special cases, you can create a separate, limited Ignition config to pass to the live system. That Ignition config could do a certain set of tasks, such as reporting success to a provisioning system after completing installation. This special Ignition config is consumed by the coreos-installer to be applied on first boot of the installed system. Do not provide the standard control plane and compute node Ignition configs to the live ISO directly.

• coreos-installer: You can boot the live ISO installer to a shell prompt, which allows you to prepare the permanent system in a variety of ways before first boot. In particular, you can run the coreos-installer command to identify various artifacts to include, work with disk partitions, and set up networking. In some cases, you can configure features on the live system and copy them to the installed system.

Whether to use an ISO or PXE install depends on your situation. A PXE install requires an available DHCP service and more preparation, but can make the installation process more automated. An ISO install is a more manual process and can be inconvenient if you are setting up more than a few machines.

Installing FCOS by using an ISO image

You can use an ISO image to install FCOS on the machines.

Prerequisites

• You have created the Ignition config files for your cluster.

• You have configured suitable network, DNS and load balancing infrastructure.

• You have an HTTP server that can be accessed from your computer, and from the machines that you create.

• You have reviewed the Advanced FCOS installation configuration section for different ways to configure features, such as networking and disk partitioning.

Procedure

1. Obtain the SHA512 digest for each of your Ignition config files. For example, you can use the following on a system running Linux to get the SHA512 digest for your bootstrap.ign Ignition config file:

   $ sha512sum <installation_directory>/bootstrap.ign

   Example output

   a5a2d43879223273c9b60af66b44202a1d1248fc01cf156c46d4a79f552b6bad47bc8cc78ddf0116e80c59d2ea9e32ba53bc807afbca581aa059311def2c3e3b  installation_directory/bootstrap.ign

   The digests are provided to the coreos-installer in a later step to validate the authenticity of the Ignition config files on the cluster nodes.

2. Upload the bootstrap, control plane, and compute node Ignition config files that the installation program created to your HTTP server. Note the URLs of these files.

   You can add or change configuration settings in your Ignition configs before saving them to your HTTP server. If you plan to add more compute machines to your cluster after you finish installation, do not delete these files.

3. From the installation host, validate that the Ignition config files are available on the URLs. The following example gets the Ignition config file for the bootstrap node:

   $ curl -k http://<HTTP_server>/bootstrap.ign (1)

   Example output

     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                    Dload  Upload   Total   Spent    Left  Speed
     0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0{"ignition":{"version":"3.2.0"},"passwd":{"users":[{"name":"core","sshAuthorizedKeys":["ssh-rsa...

   Replace bootstrap.ign with master.ign or worker.ign in the command to validate that the Ignition config files for the control plane and compute nodes are also available.

4. Obtain the FCOS images from the FCOS Downloads page

5. Use the ISO to start the FCOS installation. Use one of the following installation options:

   • Burn the ISO image to a disk and boot it directly.

   • Use ISO redirection by using a lights-out management (LOM) interface.

6. Boot the FCOS ISO image without specifying any options or interrupting the live boot sequence. Wait for the installer to boot into a shell prompt in the FCOS live environment.

   It is possible to interrupt the FCOS installation boot process to add kernel arguments. However, for this ISO procedure you should use the coreos-installer command as outlined in the following steps, instead of adding kernel arguments.

7. Run the coreos-installer command and specify the options that meet your installation requirements. At a minimum, you must specify the URL that points to the Ignition config file for the node type, and the device that you are installing to:

   $ sudo coreos-installer install --ignition-url=http://<HTTP_server>/<node_type>.ign <device> --ignition-hash=SHA512-<digest> (1) (2)

   1	You must run the coreos-installer command by using sudo, because the core user does not have the required root privileges to perform the installation.
   2	The —ignition-hash option is required when the Ignition config file is obtained through an HTTP URL to validate the authenticity of the Ignition config file on the cluster node. <digest> is the Ignition config file SHA512 digest obtained in a preceding step.

   If you want to provide your Ignition config files through an HTTPS server that uses TLS, you can add the internal certificate authority (CA) to the system trust store before running coreos-installer.

   The following example initializes a bootstrap node installation to the /dev/sda device. The Ignition config file for the bootstrap node is obtained from an HTTP web server with the IP address 192.168.1.2:

   $ sudo coreos-installer install --ignition-url=http://192.168.1.2:80/installation_directory/bootstrap.ign /dev/sda --ignition-hash=SHA512-a5a2d43879223273c9b60af66b44202a1d1248fc01cf156c46d4a79f552b6bad47bc8cc78ddf0116e80c59d2ea9e32ba53bc807afbca581aa059311def2c3e3b

8. Monitor the progress of the FCOS installation on the console of the machine.

   Be sure that the installation is successful on each node before commencing with the OKD installation. Observing the installation process can also help to determine the cause of FCOS installation issues that might arise.

9. After FCOS installs, the system reboots. During the system reboot, it applies the Ignition config file that you specified.

10. Continue to create the other machines for your cluster.

    You must create the bootstrap and control plane machines at this time. If the control plane machines are not made schedulable, also create at least two compute machines before you install OKD.

    If the required network, DNS, and load balancer infrastructure are in place, the OKD bootstrap process begins automatically after the FCOS nodes have rebooted.

    FCOS nodes do not include a default password for the core user. You can access the nodes by running ssh core@<node>.<cluster_name>.<base_domain> as a user with access to the SSH private key that is paired to the public key that you specified in your install_config.yaml file. OKD 4 cluster nodes running FCOS are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. However, when investigating installation issues, if the OKD API is not available, or the kubelet is not properly functioning on a target node, SSH access might be required for debugging or disaster recovery.

Installing FCOS by using PXE or iPXE booting

You can use PXE or iPXE booting to install FCOS on the machines.

Prerequisites

• You have created the Ignition config files for your cluster.

• You have configured suitable network, DNS and load balancing infrastructure.

• You have configured suitable PXE or iPXE infrastructure.

• You have an HTTP server that can be accessed from your computer, and from the machines that you create.

• You have reviewed the Advanced FCOS installation configuration section for different ways to configure features, such as networking and disk partitioning.

Procedure

1. Upload the bootstrap, control plane, and compute node Ignition config files that the installation program created to your HTTP server. Note the URLs of these files.

   You can add or change configuration settings in your Ignition configs before saving them to your HTTP server. If you plan to add more compute machines to your cluster after you finish installation, do not delete these files.

2. From the installation host, validate that the Ignition config files are available on the URLs. The following example gets the Ignition config file for the bootstrap node:

   $ curl -k http://<HTTP_server>/bootstrap.ign (1)

   Example output

     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                    Dload  Upload   Total   Spent    Left  Speed
     0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0{"ignition":{"version":"3.2.0"},"passwd":{"users":[{"name":"core","sshAuthorizedKeys":["ssh-rsa...

   Replace bootstrap.ign with master.ign or worker.ign in the command to validate that the Ignition config files for the control plane and compute nodes are also available.

3. Obtain the FCOS kernel, initramfs and rootfs files from the FCOS Downloads page

4. Upload the rootfs, kernel, and initramfs files to your HTTP server.

   If you plan to add more compute machines to your cluster after you finish installation, do not delete these files.

5. Configure the network boot infrastructure so that the machines boot from their local disks after FCOS is installed on them.

6. Configure PXE or iPXE installation for the FCOS images and begin the installation.

   Modify one of the following example menu entries for your environment and verify that the image and Ignition files are properly accessible:

   • For PXE:

     DEFAULT pxeboot
     TIMEOUT 20
     PROMPT 0
     LABEL pxeboot
         KERNEL http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> (1)
         APPEND initrd=http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/bootstrap.ign (2) (3)

     1	Specify the location of the live kernel file that you uploaded to your HTTP server. The URL must be HTTP, TFTP, or FTP; HTTPS and NFS are not supported.
     2	If you use multiple NICs, specify a single interface in the ip option. For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.
     3	Specify the locations of the FCOS files that you uploaded to your HTTP server. The initrd parameter value is the location of the initramfs file, the coreos.live.rootfs_url parameter value is the location of the rootfs file, and the coreos.inst.ignition_url parameter value is the location of the bootstrap Ignition config file. You can also add more kernel arguments to the APPEND line to configure networking or other boot options.

     This configuration does not enable serial console access on machines with a graphical console. To configure a different console, add one or more console= arguments to the APPEND line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux?.

   • For iPXE:

     kernel http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> initrd=main coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/bootstrap.ign (1) (2)
     initrd --name main http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img (3)
     boot

     1	Specify the locations of the FCOS files that you uploaded to your HTTP server. The kernel parameter value is the location of the kernel file, the initrd=main argument is needed for booting on UEFI systems, the coreos.live.rootfs_url parameter value is the location of the rootfs file, and the coreos.inst.ignition_url parameter value is the location of the bootstrap Ignition config file.
     2	If you use multiple NICs, specify a single interface in the ip option. For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.
     3	Specify the location of the initramfs file that you uploaded to your HTTP server.

     This configuration does not enable serial console access on machines with a graphical console. To configure a different console, add one or more console= arguments to the kernel line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux?.

7. Monitor the progress of the FCOS installation on the console of the machine.

   Be sure that the installation is successful on each node before commencing with the OKD installation. Observing the installation process can also help to determine the cause of FCOS installation issues that might arise.

8. After FCOS installs, the system reboots. During reboot, the system applies the Ignition config file that you specified.

9. Continue to create the machines for your cluster.

   You must create the bootstrap and control plane machines at this time. If the control plane machines are not made schedulable, also create at least two compute machines before you install the cluster.

   If the required network, DNS, and load balancer infrastructure are in place, the OKD bootstrap process begins automatically after the FCOS nodes have rebooted.

   FCOS nodes do not include a default password for the core user. You can access the nodes by running ssh core@<node>.<cluster_name>.<base_domain> as a user with access to the SSH private key that is paired to the public key that you specified in your install_config.yaml file. OKD 4 cluster nodes running FCOS are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. However, when investigating installation issues, if the OKD API is not available, or the kubelet is not properly functioning on a target node, SSH access might be required for debugging or disaster recovery.

Advanced FCOS installation configuration

A key benefit for manually provisioning the Fedora CoreOS (FCOS) nodes for OKD is to be able to do configuration that is not available through default OKD installation methods. This section describes some of the configurations that you can do using techniques that include:

• Passing kernel arguments to the live installer

• Running coreos-installer manually from the live system

• Embedding Ignition configs in an ISO

The advanced configuration topics for manual Fedora CoreOS (FCOS) installations detailed in this section relate to disk partitioning, networking, and using Ignition configs in different ways.

Using advanced networking options for PXE and ISO installations

Networking for OKD nodes uses DHCP by default to gather all necessary configuration settings. To set up static IP addresses or configure special settings, such as bonding, you can do one of the following:

• Pass special kernel parameters when you boot the live installer.

• Use a machine config to copy networking files to the installed system.

• Configure networking from a live installer shell prompt, then copy those settings to the installed system so that they take effect when the installed system first boots.

To configure a PXE or iPXE installation, use one of the following options:

• See the “Advanced RHCOS installation reference” tables.

• Use a machine config to copy networking files to the installed system.

To configure an ISO installation, use the following procedure.

Procedure

1. Boot the ISO installer.

2. From the live system shell prompt, configure networking for the live system using available RHEL tools, such as nmcli or nmtui.

3. Run the coreos-installer command to install the system, adding the --copy-network option to copy networking configuration. For example:

   $ coreos-installer install --copy-network \
        --ignition-url=http://host/worker.ign /dev/sda

   The —copy-network option only copies networking configuration found under /etc/NetworkManager/system-connections. In particular, it does not copy the system hostname.

4. Reboot into the installed system.

Additional resources

• See Getting started with nmcli and Getting started with nmtui in the Fedora 8 documentation for more information about the nmcli and nmtui tools.

Disk partitioning

The disk partitions are created on OKD cluster nodes during the Fedora CoreOS (FCOS) installation. Each FCOS node of a particular architecture uses the same partition layout, unless the default partitioning configuration is overridden. During the FCOS installation, the size of the root file system is increased to use the remaining available space on the target device.

There are two cases where you might want to override the default partitioning when installing FCOS on an OKD cluster node:

• Creating separate partitions: For greenfield installations on an empty disk, you might want to add separate storage to a partition. This is officially supported for mounting /var or a subdirectory of /var, such as /var/lib/etcd, on a separate partition, but not both.

  Kubernetes supports only two file system partitions. If you add more than one partition to the original configuration, Kubernetes cannot monitor all of them.

• Retaining existing partitions: For a brownfield installation where you are reinstalling OKD on an existing node and want to retain data partitions installed from your previous operating system, there are both boot arguments and options to coreos-installer that allow you to retain existing data partitions.

Creating a separate /var partition

In general, you should use the default disk partitioning that is created during the FCOS installation. However, there are cases where you might want to create a separate partition for a directory that you expect to grow.

OKD supports the addition of a single partition to attach storage to either the /var directory or a subdirectory of /var. For example:

• /var/lib/containers: Holds container-related content that can grow as more images and containers are added to a system.

• /var/lib/etcd: Holds data that you might want to keep separate for purposes such as performance optimization of etcd storage.

• /var: Holds data that you might want to keep separate for purposes such as auditing.

Storing the contents of a /var directory separately makes it easier to grow storage for those areas as needed and reinstall OKD at a later date and keep that data intact. With this method, you will not have to pull all your containers again, nor will you have to copy massive log files when you update systems.

The use of a separate partition for the /var directory or a subdirectory of /var also prevents data growth in the partitioned directory from filling up the root file system.

The following procedure sets up a separate /var partition by adding a machine config manifest that is wrapped into the Ignition config file for a node type during the preparation phase of an installation.

Procedure

1. On your installation host, change to the directory that contains the OKD installation program and generate the Kubernetes manifests for the cluster:

   $ openshift-install create manifests --dir <installation_directory>

2. Create a Butane config that configures the additional partition. For example, name the file $HOME/clusterconfig/98-var-partition.bu, change the disk device name to the name of the storage device on the worker systems, and set the storage size as appropriate. This example places the /var directory on a separate partition:

   variant: openshift
   version: 4.9.0
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 98-var-partition
   storage:
     disks:
     - device: /dev/<device_name> (1)
       partitions:
       - label: var
         start_mib: <partition_start_offset> (2)
         size_mib: <partition_size> (3)
     filesystems:
       - device: /dev/disk/by-partlabel/var
         path: /var
         format: xfs
         mount_options: [defaults, prjquota] (4)
         with_mount_unit: true

   1	The storage device name of the disk that you want to partition.
   2	When adding a data partition to the boot disk, a minimum offset value of 25000 mebibytes is recommended. The root file system is automatically resized to fill all available space up to the specified offset. If no offset value is specified, or if the specified value is smaller than the recommended minimum, the resulting root file system will be too small, and future reinstalls of FCOS might overwrite the beginning of the data partition.
   3	The size of the data partition in mebibytes.
   4	The prjquota mount option must be enabled for filesystems used for container storage.

   When creating a separate /var partition, you cannot use different instance types for compute nodes, if the different instance types do not have the same device name.

3. Create a manifest from the Butane config and save it to the clusterconfig/openshift directory. For example, run the following command:

   $ butane $HOME/clusterconfig/98-var-partition.bu -o $HOME/clusterconfig/openshift/98-var-partition.yaml

4. Create the Ignition config files:

   $ openshift-install create ignition-configs --dir <installation_directory> (1)

   1	For <installation_directory>, specify the same installation directory.

   Ignition config files are created for the bootstrap, control plane, and compute nodes in the installation directory:

   .
   ├── auth
   │   ├── kubeadmin-password
   │   └── kubeconfig
   ├── bootstrap.ign
   ├── master.ign
   ├── metadata.json
   └── worker.ign

   The files in the <installation_directory>/manifest and <installation_directory>/openshift directories are wrapped into the Ignition config files, including the file that contains the 98-var-partition custom MachineConfig object.

Next steps

• You can apply the custom disk partitioning by referencing the Ignition config files during the FCOS installations.

Retaining existing partitions

For an ISO installation, you can add options to the coreos-installer command that cause the installer to maintain one or more existing partitions. For a PXE installation, you can add coreos.inst.* options to the APPEND parameter to preserve partitions.

Saved partitions might be data partitions from an existing OKD system. You can identify the disk partitions you want to keep either by partition label or by number.

Retaining existing partitions during an ISO installation

This example preserves any partition in which the partition label begins with data (data*):

# coreos-installer install --ignition-url http://10.0.2.2:8080/user.ign \
        --save-partlabel 'data*' /dev/sda

The following example illustrates running the coreos-installer in a way that preserves the sixth (6) partition on the disk:

# coreos-installer install --ignition-url http://10.0.2.2:8080/user.ign \
        --save-partindex 6 /dev/sda

This example preserves partitions 5 and higher:

# coreos-installer install --ignition-url http://10.0.2.2:8080/user.ign
        --save-partindex 5- /dev/sda

In the previous examples where partition saving is used, coreos-installer recreates the partition immediately.

Retaining existing partitions during a PXE installation

This APPEND option preserves any partition in which the partition label begins with ‘data’ (‘data*‘):

coreos.inst.save_partlabel=data*

This APPEND option preserves partitions 5 and higher:

coreos.inst.save_partindex=5-

This APPEND option preserves partition 6:

coreos.inst.save_partindex=6

Identifying Ignition configs

When doing an FCOS manual installation, there are two types of Ignition configs that you can provide, with different reasons for providing each one:

• Permanent install Ignition config: Every manual FCOS installation needs to pass one of the Ignition config files generated by openshift-installer, such as bootstrap.ign, master.ign and worker.ign, to carry out the installation.

  It is not recommended to modify these Ignition config files directly. You can update the manifest files that are wrapped into the Ignition config files, as outlined in examples in the preceding sections.

  For PXE installations, you pass the Ignition configs on the APPEND line using the coreos.inst.ignition_url= option. For ISO installations, after the ISO boots to the shell prompt, you identify the Ignition config on the coreos-installer command line with the --ignition-url= option. In both cases, only HTTP and HTTPS protocols are supported.

• Live install Ignition config: This type must be created manually and should be avoided if possible, as it is not supported by Red Hat. With this method, the Ignition config passes to the live install medium, runs immediately upon booting, and performs setup tasks before and/or after the FCOS system installs to disk. This method should only be used for performing tasks that must be done once and not applied again later, such as with advanced partitioning that cannot be done using a machine config.

  For PXE or ISO boots, you can create the Ignition config and APPEND the ignition.config.url= option to identify the location of the Ignition config. You also need to append ignition.firstboot ignition.platform.id=metal or the ignition.config.url option will be ignored.

Embedding a live install Ignition config in the FCOS ISO

You can embed a live install Ignition config directly in an FCOS ISO image. When the ISO image is booted, the embedded config will be applied automatically.

Procedure

1. Download the coreos-installer binary from the following image mirror page: https://mirror.openshift.com/pub/openshift-v4/clients/coreos-installer/latest/.

2. Retrieve the FCOS ISO image and the Ignition config file, and copy them into an accessible directory, such as /mnt:

   # cp rhcos-<version>-live.x86_64.iso bootstrap.ign /mnt/
   # chmod 644 /mnt/rhcos-<version>-live.x86_64.iso

3. Run the following command to embed the Ignition config into the ISO:

   # ./coreos-installer iso ignition embed -i /mnt/bootstrap.ign \
        /mnt/rhcos-<version>-live.x86_64.iso

   You can now use that ISO to install FCOS using the specified live install Ignition config.

   Using coreos-installer iso ignition embed to embed a file generated by openshift-installer, such as bootstrap.ign, master.ign and worker.ign, is unsupported and not recommended.

4. To show the contents of the embedded Ignition config and direct it into a file, run:

   # ./coreos-installer iso ignition show /mnt/rhcos-<version>-live.x86_64.iso > mybootstrap.ign

   # diff -s bootstrap.ign mybootstrap.ign

   Example output

   Files bootstrap.ign and mybootstrap.ign are identical

5. To remove the Ignition config and return the ISO to its pristine state so you can reuse it, run:

   # ./coreos-installer iso ignition remove /mnt/rhcos-<version>-live.x86_64.iso

   You can now embed another Ignition config into the ISO or use the ISO in its pristine state.

Advanced FCOS installation reference

This section illustrates the networking configuration and other advanced options that allow you to modify the Fedora CoreOS (FCOS) manual installation process. The following tables describe the kernel arguments and command-line options you can use with the FCOS live installer and the coreos-installer command.

Networking and bonding options for ISO installations

If you install FCOS from an ISO image, you can add kernel arguments manually when you boot the image to configure networking for a node. If no networking arguments are specified, DHCP is activated in the initramfs when FCOS detects that networking is required to fetch the Ignition config file.

The following table provides examples for configuring networking and bonding on your FCOS nodes for ISO installations. The examples describe how to use the ip=, nameserver=, and bond= kernel arguments.

The networking options are passed to the dracut tool during system boot. For more information about the networking options supported by dracut, see the dracut.cmdline manual page.

ip=10.10.10.2::10.10.10.254:255.255.255.0:core0.example.com:enp1s0:none
nameserver=4.4.4.41

ip=10.10.10.2::10.10.10.254:255.255.255.0:core0.example.com:enp1s0:none
ip=10.10.10.3::10.10.10.254:255.255.255.0:core0.example.com:enp2s0:none

ip=::10.10.10.254::::

rd.route=20.20.20.0/24:20.20.20.254:enp2s0

ip=10.10.10.2::10.10.10.254:255.255.255.0:core0.example.com:enp1s0:none
ip=::::core0.example.com:enp2s0:none

ip=enp1s0:dhcp
ip=10.10.10.2::10.10.10.254:255.255.255.0:core0.example.com:enp2s0:none

ip=10.10.10.2::10.10.10.254:255.255.255.0:core0.example.com:enp2s0.100:none
vlan=enp2s0.100:enp2s0

ip=enp2s0.100:dhcp
vlan=enp2s0.100:enp2s0

nameserver=1.1.1.1
nameserver=8.8.8.8

bond=bond0:em1,em2:mode=active-backup
ip=bond0:dhcp

bond=bond0:em1,em2:mode=active-backup
ip=10.10.10.2::10.10.10.254:255.255.255.0:core0.example.com:bond0:none

ip=bond0.100:dhcp
bond=bond0:em1,em2:mode=active-backup
vlan=bond0.100:bond0

ip=10.10.10.2::10.10.10.254:255.255.255.0:core0.example.com:bond0.100:none
bond=bond0:em1,em2:mode=active-backup
vlan=bond0.100:bond0

team=team0:em1,em2
ip=team0:dhcp

coreos-installer options for ISO installations

You can install FCOS by running coreos-installer install <options> <device> at the command prompt, after booting into the FCOS live environment from an ISO image.

The following table shows the subcommands, options, and arguments you can pass to the coreos-installer command.

coreos.inst boot options for ISO or PXE installations

You can automatically invoke coreos-installer options at boot time by passing coreos.inst boot arguments to the FCOS live installer. These are provided in addition to the standard boot arguments.

• For ISO installations, the coreos.inst options can be added by interrupting the automatic boot at the bootloader menu. You can interrupt the automatic boot by pressing TAB while the RHEL CoreOS (Live) menu option is highlighted.

• For PXE or iPXE installations, the coreos.inst options must be added to the APPEND line before the FCOS live installer is booted.

The following table shows the FCOS live installer coreos.inst boot options for ISO and PXE installations.

Enabling multipathing with kernel arguments on FCOS

FCOS supports multipathing on the primary disk, allowing stronger resilience to hardware failure to achieve higher host availability.

You can enable multipathing at installation time for nodes that were provisioned in OKD 4.8 or later. While post-installation support is available by activating multipathing via the machine config, enabling multipathing during installation is recommended.

In setups where any I/O to non-optimized paths results in I/O system errors, you must enable multipathing at installation time.

The following procedure enables multipath at installation time and appends kernel arguments to the coreos-installer install command so that the installed system itself will use multipath beginning from the first boot.

Prerequisites

• You have a running OKD cluster that uses version 4.8 or later.

• You are logged in to the cluster as a user with administrative privileges.

Procedure

1. To enable multipath and start the multipathd daemon, run the following command:

   $ mpathconf --enable && systemctl start multipathd.service

   • Optional: If booting the PXE or ISO, you can instead enable multipath by adding rd.multipath=default from the kernel command line.

2. Append the kernel arguments by invoking the coreos-installer program:

   • If there is only one multipath device connected to the machine, it should be available at path /dev/mapper/mpatha. For example:

     $ coreos-installer install /dev/mapper/mpatha \ (1)
     --append-karg rd.multipath=default \
     --append-karg root=/dev/disk/by-label/dm-mpath-root

     1	Indicates the path of the single multipathed device.

   • If there are multiple multipath devices connected to the machine, or to be more explicit, instead of using /dev/mapper/mpatha, it is recommended to use the World Wide Name (WWN) symlink available in /dev/disk/by-id. For example:

     $ coreos-installer install /dev/disk/by-id/wwn-<wwn_ID> \ (1)
     --append-karg rd.multipath=default \
     --append-karg root=/dev/disk/by-label/dm-mpath-root

     1	Indicates the WWN ID of the target multipathed device. For example, 0xx194e957fcedb4841.

     This symlink can also be used as the coreos.inst.install_dev kernel argument when using special coreos.inst.* arguments to direct the live installer. For more information, see “Installing FCOS and starting the OKD bootstrap process”.

3. Check that the kernel arguments worked by going to one of the worker nodes and listing the kernel command line arguments (in /proc/cmdline on the host):

   $ oc debug node/ip-10-0-141-105.ec2.internal

   Example output

   Starting pod/ip-10-0-141-105ec2internal-debug ...
   To use host binaries, run `chroot /host`
   sh-4.2# cat /host/proc/cmdline
   ...
   rd.multipath=default root=/dev/disk/by-label/dm-mpath-root
   ...
   sh-4.2# exit

   You should see the added kernel arguments.

Updating the bootloader using bootupd

To update the bootloader by using bootupd, you must either install bootupd on FCOS machines manually or provide a machine config with the enabled systemd unit. Unlike grubby or other bootloader tools, bootupd does not manage kernel space configuration such as passing kernel arguments.

After you have installed bootupd, you can manage it remotely from the OKD cluster.

Manual install method

You can manually install bootupd by using the bootctl command-line tool.

1. Inspect the system status:

   # bootupctl status

   Example output

   Component EFI
     Installed: grub2-efi-x64-1:2.04-31.fc33.x86_64,shim-x64-15-8.x86_64
     Update: At latest version

2. FCOS images created without bootupd installed on them require an explicit adoption phase.

   If the system status is Adoptable, perform the adoption:

   # bootupctl adopt-and-update

   Example output

   Updated: grub2-efi-x64-1:2.04-31.fc33.x86_64,shim-x64-15-8.x86_64

3. If an update is available, apply the update so that the changes take effect on the next reboot:

   # bootupctl update

   Example output

   Updated: grub2-efi-x64-1:2.04-31.fc33.x86_64,shim-x64-15-8.x86_64

Machine config method

Another way to enable bootupd is by providing a machine config.

• Provide a machine config file with the enabled systemd unit, as shown in the following example:

  Example output

    variant: rhcos
    version: 1.1.0
    systemd:
      units:
        - name: custom-bootupd-auto.service
          enabled: true
          contents: |
            [Unit]
            Description=Bootupd automatic update
            [Service]
            ExecStart=/usr/bin/bootupctl update
            RemainAfterExit=yes
            [Install]
            WantedBy=multi-user.target