Configuring a private cluster

After you install an OKD version 4 cluster, you can set some of its core components to be private.

About private clusters

By default, OKD is provisioned using publicly-accessible DNS and endpoints. You can set the DNS, Ingress Controller, and API server to private after you deploy your private cluster.

If the cluster has any public subnets, load balancer services created by administrators might be publicly accessible. To ensure cluster security, verify that these services are explicitly annotated as private.

DNS

If you install OKD on installer-provisioned infrastructure, the installation program creates records in a pre-existing public zone and, where possible, creates a private zone for the cluster’s own DNS resolution. In both the public zone and the private zone, the installation program or cluster creates DNS entries for *.apps, for the Ingress object, and api, for the API server.

The *.apps records in the public and private zone are identical, so when you delete the public zone, the private zone seamlessly provides all DNS resolution for the cluster.

Ingress Controller

Because the default Ingress object is created as public, the load balancer is internet-facing and in the public subnets.

The Ingress Operator generates a default certificate for an Ingress Controller to serve as a placeholder until you configure a custom default certificate. Do not use Operator-generated default certificates in production clusters. The Ingress Operator does not rotate its own signing certificate or the default certificates that it generates. Operator-generated default certificates are intended as placeholders for custom default certificates that you configure.

API server

By default, the installation program creates appropriate network load balancers for the API server to use for both internal and external traffic.

On Amazon Web Services (AWS), separate public and private load balancers are created. The load balancers are identical except that an additional port is available on the internal one for use within the cluster. Although the installation program automatically creates or destroys the load balancer based on API server requirements, the cluster does not manage or maintain them. As long as you preserve the cluster’s access to the API server, you can manually modify or move the load balancers. For the public load balancer, port 6443 is open and the health check is configured for HTTPS against the /readyz path.

On Google Cloud Platform, a single load balancer is created to manage both internal and external API traffic, so you do not need to modify the load balancer.

On Microsoft Azure, both public and private load balancers are created. However, because of limitations in current implementation, you just retain both load balancers in a private cluster.

Setting DNS to private

After you deploy a cluster, you can modify its DNS to use only a private zone.

Procedure

  1. Review the DNS custom resource for your cluster:

    1. $ oc get dnses.config.openshift.io/cluster -o yaml

    Example output

    1. apiVersion: config.openshift.io/v1
    2. kind: DNS
    3. metadata:
    4.   creationTimestamp: "2019-10-25T18:27:09Z"
    5.   generation: 2
    6.   name: cluster
    7.   resourceVersion: "37966"
    8.   selfLink: /apis/config.openshift.io/v1/dnses/cluster
    9.   uid: 0e714746-f755-11f9-9cb1-02ff55d8f976
    10. spec:
    11.   baseDomain: <base_domain>
    12.   privateZone:
    13.     tags:
    14.       Name: <infrastructure_id>-int
    15.       kubernetes.io/cluster/<infrastructure_id>: owned
    16.   publicZone:
    17.     id: Z2XXXXXXXXXXA4
    18. status: {}

    Note that the spec section contains both a private and a public zone.

  2. Patch the DNS custom resource to remove the public zone:

    1. $ oc patch dnses.config.openshift.io/cluster --type=merge --patch='{"spec": {"publicZone": null}}'
    2. dns.config.openshift.io/cluster patched

    Because the Ingress Controller consults the DNS definition when it creates Ingress objects, when you create or modify Ingress objects, only private records are created.

    DNS records for the existing Ingress objects are not modified when you remove the public zone.

  3. Optional: Review the DNS custom resource for your cluster and confirm that the public zone was removed:

    1. $ oc get dnses.config.openshift.io/cluster -o yaml

    Example output

    1. apiVersion: config.openshift.io/v1
    2. kind: DNS
    3. metadata:
    4.   creationTimestamp: "2019-10-25T18:27:09Z"
    5.   generation: 2
    6.   name: cluster
    7.   resourceVersion: "37966"
    8.   selfLink: /apis/config.openshift.io/v1/dnses/cluster
    9.   uid: 0e714746-f755-11f9-9cb1-02ff55d8f976
    10. spec:
    11.   baseDomain: <base_domain>
    12.   privateZone:
    13.     tags:
    14.       Name: <infrastructure_id>-int
    15.       kubernetes.io/cluster/<infrastructure_id>-wfpg4: owned
    16. status: {}

Setting the Ingress Controller to private

After you deploy a cluster, you can modify its Ingress Controller to use only a private zone.

Procedure

  1. Modify the default Ingress Controller to use only an internal endpoint:

    1. $ oc replace --force --wait --filename - <<EOF
    2. apiVersion: operator.openshift.io/v1
    3. kind: IngressController
    4. metadata:
    5.   namespace: openshift-ingress-operator
    6.   name: default
    7. spec:
    8.   endpointPublishingStrategy:
    9.     type: LoadBalancerService
    10.     loadBalancer:
    11.       scope: Internal
    12. EOF

    Example output

    1. ingresscontroller.operator.openshift.io "default" deleted
    2. ingresscontroller.operator.openshift.io/default replaced

    The public DNS entry is removed, and the private zone entry is updated.

Restricting the API server to private

After you deploy a cluster to Amazon Web Services (AWS) or Microsoft Azure, you can reconfigure the API server to use only the private zone.

Prerequisites

  • Install the OpenShift CLI (oc).

  • Have access to the web console as a user with admin privileges.

Procedure

  1. In the web portal or console for your cloud provider, take the following actions:

    1. Locate and delete the appropriate load balancer component:

      • For AWS, delete the external load balancer. The API DNS entry in the private zone already points to the internal load balancer, which uses an identical configuration, so you do not need to modify the internal load balancer.

      • For Azure, delete the api-internal rule for the load balancer.

    2. Delete the api.$clustername.$yourdomain DNS entry in the public zone.

  2. Remove the external load balancers:

    You can run the following steps only for an installer-provisioned infrastructure (IPI) cluster. For a user-provisioned infrastructure (UPI) cluster, you must manually remove or disable the external load balancers.

    • If your cluster uses a control plane machine set, delete the following lines in the control plane machine set custom resource:

      1. providerSpec:
      2.   value:
      3.     loadBalancers:
      4.     - name: lk4pj-ext (1)
      5.       type: network (1)
      6.     - name: lk4pj-int
      7.       type: network
      1Delete this line.

    • If your cluster does not use a control plane machine set, you must delete the external load balancers from each control plane machine.

      1. From your terminal, list the cluster machines by running the following command:

        1. $ oc get machine -n openshift-machine-api

        Example output

        1. NAME                            STATE     TYPE        REGION      ZONE         AGE
        2. lk4pj-master-0                  running   m4.xlarge   us-east-1   us-east-1a   17m
        3. lk4pj-master-1                  running   m4.xlarge   us-east-1   us-east-1b   17m
        4. lk4pj-master-2                  running   m4.xlarge   us-east-1   us-east-1a   17m
        5. lk4pj-worker-us-east-1a-5fzfj   running   m4.xlarge   us-east-1   us-east-1a   15m
        6. lk4pj-worker-us-east-1a-vbghs   running   m4.xlarge   us-east-1   us-east-1a   15m
        7. lk4pj-worker-us-east-1b-zgpzg   running   m4.xlarge   us-east-1   us-east-1b   15m

        The control plane machines contain master in the name.

      2. Remove the external load balancer from each control plane machine:

        1. Edit a control plane machine object to by running the following command:

          1. $ oc edit machines -n openshift-machine-api <control_plane_name> (1)
          1Specify the name of the control plane machine object to modify.

        2. Remove the lines that describe the external load balancer, which are marked in the following example:

          1. providerSpec:
          2.   value:
          3.     loadBalancers:
          4.     - name: lk4pj-ext (1)
          5.       type: network (1)
          6.     - name: lk4pj-int
          7.       type: network
          1Delete this line.

        3. Save your changes and exit the object specification.

        4. Repeat this process for each of the control plane machines.

Configuring the Ingress Controller endpoint publishing scope to Internal

When a cluster administrator installs a new cluster without specifying that the cluster is private, the default Ingress Controller is created with a scope set to External. Cluster administrators can change an External scoped Ingress Controller to Internal.

Prerequisites

  • You installed the oc CLI.

Procedure

  • To change an External scoped Ingress Controller to Internal, enter the following command:

    1. $ oc -n openshift-ingress-operator patch ingresscontrollers/default --type=merge --patch='{"spec":{"endpointPublishingStrategy":{"type":"LoadBalancerService","loadBalancer":{"scope":"Internal"}}}}'

  • To check the status of the Ingress Controller, enter the following command:

    1. $ oc -n openshift-ingress-operator get ingresscontrollers/default -o yaml

    • The Progressing status condition indicates whether you must take further action. For example, the status condition can indicate that you need to delete the service by entering the following command:

      1. $ oc -n openshift-ingress delete services/router-default

      If you delete the service, the Ingress Operator recreates it as Internal.

Configuring a private storage endpoint on Azure

You can leverage the Image Registry Operator to use private endpoints on Azure, which enables seamless configuration of private storage accounts when OKD is deployed on private Azure clusters. This allows you to deploy the image registry without exposing public-facing storage endpoints.

You can configure the Image Registry Operator to use private storage endpoints on Azure in one of two ways:

  • By configuring the Image Registry Operator to discover the VNet and subnet names

  • With user-provided Azure Virtual Network (VNet) and subnet names

Limitations for configuring a private storage endpoint on Azure

The following limitations apply when configuring a private storage endpoint on Azure:

  • When configuring the Image Registry Operator to use a private storage endpoint, public network access to the storage account is disabled. Consequently, pulling images from the registry outside of OKD only works by setting disableRedirect: true in the registry Operator configuration. With redirect enabled, the registry redirects the client to pull images directly from the storage account, which will no longer work due to disabled public network access. For more information, see “Disabling redirect when using a private storage endpoint on Azure”.

  • This operation cannot be undone by the Image Registry Operator.

Configuring a private storage endpoint on Azure by enabling the Image Registry Operator to discover VNet and subnet names

The following procedure shows you how to set up a private storage endpoint on Azure by configuring the Image Registry Operator to discover VNet and subnet names.

Prerequisites

  • You have configured the image registry to run on Azure.

  • Your network has been set up using the Installer Provisioned Infrastructure installation method.

    For users with a custom network setup, see “Configuring a private storage endpoint on Azure with user-provided VNet and subnet names”.

Procedure

  1. Edit the Image Registry Operator config object and set networkAccess.type to Internal:

    1. $ oc edit configs.imageregistry/cluster
    1. # ...
    2. spec:
    3.   # ...
    4.    storage:
    5.       azure:
    6.         # ...
    7.         networkAccess:
    8.           type: Internal
    9. # ...

  2. Optional: Enter the following command to confirm that the Operator has completed provisioning. This might take a few minutes.

    1. $ oc get configs.imageregistry/cluster -o=jsonpath="{.spec.storage.azure.privateEndpointName}" -w

  3. Optional: If the registry is exposed by a route, and you are configuring your storage account to be private, you must disable redirect if you want pulls external to the cluster to continue to work. Enter the following command to disable redirect on the Image Operator configuration:

    1. $ oc patch configs.imageregistry cluster --type=merge -p '{"spec":{"disableRedirect": true}}'

    When redirect is enabled, pulling images from outside of the cluster will not work.

Verification

  1. Fetch the registry service name by running the following command:

    1. $ oc registry info --internal=true

    Example output

    1. image-registry.openshift-image-registry.svc:5000

  2. Enter debug mode by running the following command:

    1. $ oc debug node/<node_name>

  3. Run the suggested chroot command. For example:

    1. $ chroot /host

  4. Enter the following command to log in to your container registry:

    1. $ podman login --tls-verify=false -u unused -p $(oc whoami -t) image-registry.openshift-image-registry.svc:5000

    Example output

    1. Login Succeeded!

  5. Enter the following command to verify that you can pull an image from the registry:

    1. $ podman pull --tls-verify=false image-registry.openshift-image-registry.svc:5000/openshift/tools

    Example output

    1. Trying to pull image-registry.openshift-image-registry.svc:5000/openshift/tools/openshift/tools...
    2. Getting image source signatures
    3. Copying blob 6b245f040973 done
    4. Copying config 22667f5368 done
    5. Writing manifest to image destination
    6. Storing signatures
    7. 22667f53682a2920948d19c7133ab1c9c3f745805c14125859d20cede07f11f9

Configuring a private storage endpoint on Azure with user-provided VNet and subnet names

Use the following procedure to configure a storage account that has public network access disabled and is exposed behind a private storage endpoint on Azure.

Prerequisites

  • You have configured the image registry to run on Azure.

  • You must know the VNet and subnet names used for your Azure environment.

  • If your network was configured in a separate resource group in Azure, you must also know its name.

Procedure

  1. Edit the Image Registry Operator config object and configure the private endpoint using your VNet and subnet names:

    1. $ oc edit configs.imageregistry/cluster
    1. # ...
    2. spec:
    3.   # ...
    4.    storage:
    5.       azure:
    6.         # ...
    7.         networkAccess:
    8.           type: Internal
    9.           internal:
    10.             subnetName: <subnet_name>
    11.             vnetName: <vnet_name>
    12.             networkResourceGroupName: <network_resource_group_name>
    13. # ...

  2. Optional: Enter the following command to confirm that the Operator has completed provisioning. This might take a few minutes.

    1. $ oc get configs.imageregistry/cluster -o=jsonpath="{.spec.storage.azure.privateEndpointName}" -w

    When redirect is enabled, pulling images from outside of the cluster will not work.

Verification

  1. Fetch the registry service name by running the following command:

    1. $ oc registry info --internal=true

    Example output

    1. image-registry.openshift-image-registry.svc:5000

  2. Enter debug mode by running the following command:

    1. $ oc debug node/<node_name>

  3. Run the suggested chroot command. For example:

    1. $ chroot /host

  4. Enter the following command to log in to your container registry:

    1. $ podman login --tls-verify=false -u unused -p $(oc whoami -t) image-registry.openshift-image-registry.svc:5000

    Example output

    1. Login Succeeded!

  5. Enter the following command to verify that you can pull an image from the registry:

    1. $ podman pull --tls-verify=false image-registry.openshift-image-registry.svc:5000/openshift/tools

    Example output

    1. Trying to pull image-registry.openshift-image-registry.svc:5000/openshift/tools/openshift/tools...
    2. Getting image source signatures
    3. Copying blob 6b245f040973 done
    4. Copying config 22667f5368 done
    5. Writing manifest to image destination
    6. Storing signatures
    7. 22667f53682a2920948d19c7133ab1c9c3f745805c14125859d20cede07f11f9

Optional: Disabling redirect when using a private storage endpoint on Azure

By default, redirect is enabled when using the image registry. Redirect allows off-loading of traffic from the registry pods into the object storage, which makes pull faster. When redirect is enabled and the storage account is private, users from outside of the cluster are unable to pull images from the registry.

In some cases, users might want to disable redirect so that users from outside of the cluster can pull images from the registry.

Use the following procedure to disable redirect.

Prerequisites

  • You have configured the image registry to run on Azure.

  • You have configured a route.

Procedure

  • Enter the following command to disable redirect on the image registry configuration:

    1. $ oc patch configs.imageregistry cluster --type=merge -p '{"spec":{"disableRedirect": true}}'

Verification

  1. Fetch the registry service name by running the following command:

    1. $ oc registry info

    Example output

    1. default-route-openshift-image-registry.<cluster_dns>

  2. Enter the following command to log in to your container registry:

    1. $ podman login --tls-verify=false -u unused -p $(oc whoami -t) default-route-openshift-image-registry.<cluster_dns>

    Example output

    1. Login Succeeded!

  3. Enter the following command to verify that you can pull an image from the registry:

    1. $ podman pull --tls-verify=false default-route-openshift-image-registry.<cluster_dns>
    2. /openshift/tools

    Example output

    1. Trying to pull default-route-openshift-image-registry.<cluster_dns>/openshift/tools...
    2. Getting image source signatures
    3. Copying blob 6b245f040973 done
    4. Copying config 22667f5368 done
    5. Writing manifest to image destination
    6. Storing signatures
    7. 22667f53682a2920948d19c7133ab1c9c3f745805c14125859d20cede07f11f9