Postinstallation network configuration

After installing OKD, you can further expand and customize your network to your requirements.

Cluster Network Operator configuration

The configuration for the cluster network is specified as part of the Cluster Network Operator (CNO) configuration and stored in a custom resource (CR) object that is named cluster. The CR specifies the fields for the Network API in the operator.openshift.io API group.

The CNO configuration inherits the following fields during cluster installation from the Network API in the Network.config.openshift.io API group and these fields cannot be changed:

clusterNetwork

IP address pools from which pod IP addresses are allocated.

serviceNetwork

IP address pool for services.

defaultNetwork.type

Cluster network plugin, such as OpenShift SDN or OVN-Kubernetes.

After cluster installation, you cannot modify the fields listed in the previous section.

Enabling the cluster-wide proxy

The Proxy object is used to manage the cluster-wide egress proxy. When a cluster is installed or upgraded without the proxy configured, a Proxy object is still generated but it will have a nil spec. For example:

  1. apiVersion: config.openshift.io/v1
  2. kind: Proxy
  3. metadata:
  4.   name: cluster
  5. spec:
  6.   trustedCA:
  7.     name: ""
  8. status:

A cluster administrator can configure the proxy for OKD by modifying this cluster Proxy object.

Only the Proxy object named cluster is supported, and no additional proxies can be created.

Prerequisites

  • Cluster administrator permissions

  • OKD oc CLI tool installed

Procedure

  1. Create a config map that contains any additional CA certificates required for proxying HTTPS connections.

    You can skip this step if the proxy’s identity certificate is signed by an authority from the RHCOS trust bundle.

    1. Create a file called user-ca-bundle.yaml with the following contents, and provide the values of your PEM-encoded certificates:

      1. apiVersion: v1
      2. data:
      3.   ca-bundle.crt: | (1)
      4.     <MY_PEM_ENCODED_CERTS> (2)
      5. kind: ConfigMap
      6. metadata:
      7.   name: user-ca-bundle (3)
      8.   namespace: openshift-config (4)
      1This data key must be named ca-bundle.crt.
      2One or more PEM-encoded X.509 certificates used to sign the proxy’s identity certificate.
      3The config map name that will be referenced from the Proxy object.
      4The config map must be in the openshift-config namespace.

    2. Create the config map from this file:

      1. $ oc create -f user-ca-bundle.yaml

  2. Use the oc edit command to modify the Proxy object:

    1. $ oc edit proxy/cluster

  3. Configure the necessary fields for the proxy:

    1. apiVersion: config.openshift.io/v1
    2. kind: Proxy
    3. metadata:
    4.   name: cluster
    5. spec:
    6.   httpProxy: http://<username>:<pswd>@<ip>:<port> (1)
    7.   httpsProxy: https://<username>:<pswd>@<ip>:<port> (2)
    8.   noProxy: example.com (3)
    9.   readinessEndpoints:
    10.   - http://www.google.com (4)
    11.   - https://www.google.com
    12.   trustedCA:
    13.     name: user-ca-bundle (5)
    1A proxy URL to use for creating HTTP connections outside the cluster. The URL scheme must be http.
    2A proxy URL to use for creating HTTPS connections outside the cluster. The URL scheme must be either http or https. Specify a URL for the proxy that supports the URL scheme. For example, most proxies will report an error if they are configured to use https but they only support http. This failure message may not propagate to the logs and can appear to be a network connection failure instead. If using a proxy that listens for https connections from the cluster, you may need to configure the cluster to accept the CAs and certificates that the proxy uses.
    3A comma-separated list of destination domain names, domains, IP addresses or other network CIDRs to exclude proxying.

    Preface a domain with . to match subdomains only. For example, .y.com matches x.y.com, but not y.com. Use * to bypass proxy for all destinations. If you scale up workers that are not included in the network defined by the networking.machineNetwork[].cidr field from the installation configuration, you must add them to this list to prevent connection issues.

    This field is ignored if neither the httpProxy or httpsProxy fields are set.

    4One or more URLs external to the cluster to use to perform a readiness check before writing the httpProxy and httpsProxy values to status.
    5A reference to the config map in the openshift-config namespace that contains additional CA certificates required for proxying HTTPS connections. Note that the config map must already exist before referencing it here. This field is required unless the proxy’s identity certificate is signed by an authority from the RHCOS trust bundle.

  4. Save the file to apply the changes.

Setting DNS to private

After you deploy a cluster, you can modify its DNS to use only a private zone.

Procedure

  1. Review the DNS custom resource for your cluster:

    1. $ oc get dnses.config.openshift.io/cluster -o yaml

    Example output

    1. apiVersion: config.openshift.io/v1
    2. kind: DNS
    3. metadata:
    4.   creationTimestamp: "2019-10-25T18:27:09Z"
    5.   generation: 2
    6.   name: cluster
    7.   resourceVersion: "37966"
    8.   selfLink: /apis/config.openshift.io/v1/dnses/cluster
    9.   uid: 0e714746-f755-11f9-9cb1-02ff55d8f976
    10. spec:
    11.   baseDomain: <base_domain>
    12.   privateZone:
    13.     tags:
    14.       Name: <infrastructure_id>-int
    15.       kubernetes.io/cluster/<infrastructure_id>: owned
    16.   publicZone:
    17.     id: Z2XXXXXXXXXXA4
    18. status: {}

    Note that the spec section contains both a private and a public zone.

  2. Patch the DNS custom resource to remove the public zone:

    1. $ oc patch dnses.config.openshift.io/cluster --type=merge --patch='{"spec": {"publicZone": null}}'
    2. dns.config.openshift.io/cluster patched

    Because the Ingress Controller consults the DNS definition when it creates Ingress objects, when you create or modify Ingress objects, only private records are created.

    DNS records for the existing Ingress objects are not modified when you remove the public zone.

  3. Optional: Review the DNS custom resource for your cluster and confirm that the public zone was removed:

    1. $ oc get dnses.config.openshift.io/cluster -o yaml

    Example output

    1. apiVersion: config.openshift.io/v1
    2. kind: DNS
    3. metadata:
    4.   creationTimestamp: "2019-10-25T18:27:09Z"
    5.   generation: 2
    6.   name: cluster
    7.   resourceVersion: "37966"
    8.   selfLink: /apis/config.openshift.io/v1/dnses/cluster
    9.   uid: 0e714746-f755-11f9-9cb1-02ff55d8f976
    10. spec:
    11.   baseDomain: <base_domain>
    12.   privateZone:
    13.     tags:
    14.       Name: <infrastructure_id>-int
    15.       kubernetes.io/cluster/<infrastructure_id>-wfpg4: owned
    16. status: {}

Configuring ingress cluster traffic

OKD provides the following methods for communicating from outside the cluster with services running in the cluster:

  • If you have HTTP/HTTPS, use an Ingress Controller.

  • If you have a TLS-encrypted protocol other than HTTPS, such as TLS with the SNI header, use an Ingress Controller.

  • Otherwise, use a load balancer, an external IP, or a node port.

MethodPurpose

Use an Ingress Controller

Allows access to HTTP/HTTPS traffic and TLS-encrypted protocols other than HTTPS, such as TLS with the SNI header.

Automatically assign an external IP by using a load balancer service

Allows traffic to non-standard ports through an IP address assigned from a pool.

Manually assign an external IP to a service

Allows traffic to non-standard ports through a specific IP address.

Configure a NodePort

Expose a service on all nodes in the cluster.

Configuring the node port service range

As a cluster administrator, you can expand the available node port range. If your cluster uses of a large number of node ports, you might need to increase the number of available ports.

The default port range is 30000-32767. You can never reduce the port range, even if you first expand it beyond the default range.

Prerequisites

  • Your cluster infrastructure must allow access to the ports that you specify within the expanded range. For example, if you expand the node port range to 30000-32900, the inclusive port range of 32768-32900 must be allowed by your firewall or packet filtering configuration.

Expanding the node port range

You can expand the node port range for the cluster.

Prerequisites

  • Install the OpenShift CLI (oc).

  • Log in to the cluster with a user with cluster-admin privileges.

Procedure

  1. To expand the node port range, enter the following command. Replace <port> with the largest port number in the new range.

    1. $ oc patch network.config.openshift.io cluster --type=merge -p \
    2.   '{
    3.     "spec":
    4.       { "serviceNodePortRange": "30000-<port>" }
    5.   }'

    You can alternatively apply the following YAML to update the node port range:

    1. apiVersion: config.openshift.io/v1
    2. kind: Network
    3. metadata:
    4.   name: cluster
    5. spec:
    6.   serviceNodePortRange: “30000-<port>”

    Example output

    1. network.config.openshift.io/cluster patched

  2. To confirm that the configuration is active, enter the following command. It can take several minutes for the update to apply.

    1. $ oc get configmaps -n openshift-kube-apiserver config \
    2.   -o jsonpath="{.data['config\.yaml']}" | \
    3.   grep -Eo '"service-node-port-range":["[[:digit:]]+-[[:digit:]]+"]'

    Example output

    1. "service-node-port-range":["30000-33000"]

Configuring IPsec encryption

With IPsec enabled, all network traffic between nodes on the OVN-Kubernetes network plugin travels through an encrypted tunnel.

IPsec is disabled by default.

Prerequisites

  • Your cluster must use the OVN-Kubernetes network plugin.

Enabling pod-to-pod IPsec encryption

As a cluster administrator, you can enable pod-to-pod IPsec encryption after cluster installation.

Prerequisites

  • Install the OpenShift CLI (oc).

  • You are logged in to the cluster as a user with cluster-admin privileges.

  • You have reduced the size of your cluster MTU by 46 bytes to allow for the overhead of the IPsec ESP header.

Procedure

  • To enable IPsec encryption, enter the following command:

    1. $ oc patch networks.operator.openshift.io cluster --type=merge \
    2. -p '{"spec":{"defaultNetwork":{"ovnKubernetesConfig":{"ipsecConfig":{ }}}}}'

Verifying that IPsec is enabled

As a cluster administrator, you can verify that IPsec is enabled.

Verification

  1. To find the names of the OVN-Kubernetes data plane pods, enter the following command:

    1. $ oc get pods -n openshift-ovn-kubernetes -l=app=ovnkube-node

    Example output

    1. ovnkube-node-5xqbf                       8/8     Running   0              28m
    2. ovnkube-node-6mwcx                       8/8     Running   0              29m
    3. ovnkube-node-ck5fr                       8/8     Running   0              31m
    4. ovnkube-node-fr4ld                       8/8     Running   0              26m
    5. ovnkube-node-wgs4l                       8/8     Running   0              33m
    6. ovnkube-node-zfvcl                       8/8     Running   0              34m

  2. Verify that IPsec is enabled on your cluster:

    1. $ oc -n openshift-ovn-kubernetes -c nbdb rsh ovnkube-node-<XXXXX> ovn-nbctl --no-leader-only get nb_global . ipsec

    where:

    <XXXXX>

    Specifies the random sequence of letters for a pod from the previous step.

    Example output

    1. true

Configuring network policy

As a cluster administrator or project administrator, you can configure network policies for a project.

About network policy

In a cluster using a network plugin that supports Kubernetes network policy, network isolation is controlled entirely by NetworkPolicy objects. In OKD 4.14, OpenShift SDN supports using network policy in its default network isolation mode.

Network policy does not apply to the host network namespace. Pods with host networking enabled are unaffected by network policy rules. However, pods connecting to the host-networked pods might be affected by the network policy rules.

Network policies cannot block traffic from localhost or from their resident nodes.

By default, all pods in a project are accessible from other pods and network endpoints. To isolate one or more pods in a project, you can create NetworkPolicy objects in that project to indicate the allowed incoming connections. Project administrators can create and delete NetworkPolicy objects within their own project.

If a pod is matched by selectors in one or more NetworkPolicy objects, then the pod will accept only connections that are allowed by at least one of those NetworkPolicy objects. A pod that is not selected by any NetworkPolicy objects is fully accessible.

A network policy applies to only the TCP, UDP, and SCTP protocols. Other protocols are not affected.

The following example NetworkPolicy objects demonstrate supporting different scenarios:

  • Deny all traffic:

    To make a project deny by default, add a NetworkPolicy object that matches all pods but accepts no traffic:

    1. kind: NetworkPolicy
    2. apiVersion: networking.k8s.io/v1
    3. metadata:
    4.   name: deny-by-default
    5. spec:
    6.   podSelector: {}
    7.   ingress: []

  • Only allow connections from the OKD Ingress Controller:

    To make a project allow only connections from the OKD Ingress Controller, add the following NetworkPolicy object.

    1. apiVersion: networking.k8s.io/v1
    2. kind: NetworkPolicy
    3. metadata:
    4.   name: allow-from-openshift-ingress
    5. spec:
    6.   ingress:
    7.   - from:
    8.     - namespaceSelector:
    9.         matchLabels:
    10.           network.openshift.io/policy-group: ingress
    11.   podSelector: {}
    12.   policyTypes:
    13.   - Ingress

  • Only accept connections from pods within a project:

    To make pods accept connections from other pods in the same project, but reject all other connections from pods in other projects, add the following NetworkPolicy object:

    1. kind: NetworkPolicy
    2. apiVersion: networking.k8s.io/v1
    3. metadata:
    4.   name: allow-same-namespace
    5. spec:
    6.   podSelector: {}
    7.   ingress:
    8.   - from:
    9.     - podSelector: {}

  • Only allow HTTP and HTTPS traffic based on pod labels:

    To enable only HTTP and HTTPS access to the pods with a specific label (role=frontend in following example), add a NetworkPolicy object similar to the following:

    1. kind: NetworkPolicy
    2. apiVersion: networking.k8s.io/v1
    3. metadata:
    4.   name: allow-http-and-https
    5. spec:
    6.   podSelector:
    7.     matchLabels:
    8.       role: frontend
    9.   ingress:
    10.   - ports:
    11.     - protocol: TCP
    12.       port: 80
    13.     - protocol: TCP
    14.       port: 443

  • Accept connections by using both namespace and pod selectors:

    To match network traffic by combining namespace and pod selectors, you can use a NetworkPolicy object similar to the following:

    1. kind: NetworkPolicy
    2. apiVersion: networking.k8s.io/v1
    3. metadata:
    4.   name: allow-pod-and-namespace-both
    5. spec:
    6.   podSelector:
    7.     matchLabels:
    8.       name: test-pods
    9.   ingress:
    10.     - from:
    11.       - namespaceSelector:
    12.           matchLabels:
    13.             project: project_name
    14.         podSelector:
    15.           matchLabels:
    16.             name: test-pods

NetworkPolicy objects are additive, which means you can combine multiple NetworkPolicy objects together to satisfy complex network requirements.

For example, for the NetworkPolicy objects defined in previous samples, you can define both allow-same-namespace and allow-http-and-https policies within the same project. Thus allowing the pods with the label role=frontend, to accept any connection allowed by each policy. That is, connections on any port from pods in the same namespace, and connections on ports 80 and 443 from pods in any namespace.

Using the allow-from-router network policy

Use the following NetworkPolicy to allow external traffic regardless of the router configuration:

  1. apiVersion: networking.k8s.io/v1
  2. kind: NetworkPolicy
  3. metadata:
  4.   name: allow-from-router
  5. spec:
  6.   ingress:
  7.   - from:
  8.     - namespaceSelector:
  9.         matchLabels:
  10.           policy-group.network.openshift.io/ingress: ""(1)
  11.   podSelector: {}
  12.   policyTypes:
  13.   - Ingress
1policy-group.network.openshift.io/ingress:”” label supports both OpenShift-SDN and OVN-Kubernetes.

Using the allow-from-hostnetwork network policy

Add the following allow-from-hostnetwork NetworkPolicy object to direct traffic from the host network pods:

  1. apiVersion: networking.k8s.io/v1
  2. kind: NetworkPolicy
  3. metadata:
  4.   name: allow-from-hostnetwork
  5. spec:
  6.   ingress:
  7.   - from:
  8.     - namespaceSelector:
  9.         matchLabels:
  10.           policy-group.network.openshift.io/host-network: ""
  11.   podSelector: {}
  12.   policyTypes:
  13.   - Ingress

Example NetworkPolicy object

The following annotates an example NetworkPolicy object:

  1. kind: NetworkPolicy
  2. apiVersion: networking.k8s.io/v1
  3. metadata:
  4.   name: allow-27107 (1)
  5. spec:
  6.   podSelector: (2)
  7.     matchLabels:
  8.       app: mongodb
  9.   ingress:
  10.   - from:
  11.     - podSelector: (3)
  12.         matchLabels:
  13.           app: app
  14.     ports: (4)
  15.     - protocol: TCP
  16.       port: 27017
1The name of the NetworkPolicy object.
2A selector that describes the pods to which the policy applies. The policy object can only select pods in the project that defines the NetworkPolicy object.
3A selector that matches the pods from which the policy object allows ingress traffic. The selector matches pods in the same namespace as the NetworkPolicy.
4A list of one or more destination ports on which to accept traffic.

Creating a network policy using the CLI

To define granular rules describing ingress or egress network traffic allowed for namespaces in your cluster, you can create a network policy.

If you log in with a user with the cluster-admin role, then you can create a network policy in any namespace in the cluster.

Prerequisites

  • Your cluster uses a network plugin that supports NetworkPolicy objects, such as the OVN-Kubernetes network plugin or the OpenShift SDN network plugin with mode: NetworkPolicy set. This mode is the default for OpenShift SDN.

  • You installed the OpenShift CLI (oc).

  • You are logged in to the cluster with a user with admin privileges.

  • You are working in the namespace that the network policy applies to.

Procedure

  1. Create a policy rule:

    1. Create a <policy_name>.yaml file:

      1. $ touch <policy_name>.yaml

      where:

      <policy_name>

      Specifies the network policy file name.

    2. Define a network policy in the file that you just created, such as in the following examples:

      Deny ingress from all pods in all namespaces

      This is a fundamental policy, blocking all cross-pod networking other than cross-pod traffic allowed by the configuration of other Network Policies.

      1. kind: NetworkPolicy
      2. apiVersion: networking.k8s.io/v1
      3. metadata:
      4.   name: deny-by-default
      5. spec:
      6.   podSelector:
      7.   ingress: []

      Allow ingress from all pods in the same namespace

      1. kind: NetworkPolicy
      2. apiVersion: networking.k8s.io/v1
      3. metadata:
      4.   name: allow-same-namespace
      5. spec:
      6.   podSelector:
      7.   ingress:
      8.   - from:
      9.     - podSelector: {}

      Allow ingress traffic to one pod from a particular namespace

      This policy allows traffic to pods labelled pod-a from pods running in namespace-y.

      1. kind: NetworkPolicy
      2. apiVersion: networking.k8s.io/v1
      3. metadata:
      4.   name: allow-traffic-pod
      5. spec:
      6.   podSelector:
      7.    matchLabels:
      8.       pod: pod-a
      9.   policyTypes:
      10.   - Ingress
      11.   ingress:
      12.   - from:
      13.     - namespaceSelector:
      14.         matchLabels:
      15.            kubernetes.io/metadata.name: namespace-y

  2. To create the network policy object, enter the following command:

    1. $ oc apply -f <policy_name>.yaml -n <namespace>

    where:

    <policy_name>

    Specifies the network policy file name.

    <namespace>

    Optional: Specifies the namespace if the object is defined in a different namespace than the current namespace.

    Example output

    1. networkpolicy.networking.k8s.io/deny-by-default created

If you log in to the web console with cluster-admin privileges, you have a choice of creating a network policy in any namespace in the cluster directly in YAML or from a form in the web console.

Configuring multitenant isolation by using network policy

You can configure your project to isolate it from pods and services in other project namespaces.

Prerequisites

  • Your cluster uses a network plugin that supports NetworkPolicy objects, such as the OVN-Kubernetes network plugin or the OpenShift SDN network plugin with mode: NetworkPolicy set. This mode is the default for OpenShift SDN. This mode is the default for OpenShift SDN.

  • You installed the OpenShift CLI (oc).

  • You are logged in to the cluster with a user with admin privileges.

Procedure

  1. Create the following NetworkPolicy objects:

    1. A policy named allow-from-openshift-ingress.

      1. $ cat << EOF| oc create -f -
      2. apiVersion: networking.k8s.io/v1
      3. kind: NetworkPolicy
      4. metadata:
      5.   name: allow-from-openshift-ingress
      6. spec:
      7.   ingress:
      8.   - from:
      9.     - namespaceSelector:
      10.         matchLabels:
      11.           policy-group.network.openshift.io/ingress: ""
      12.   podSelector: {}
      13.   policyTypes:
      14.   - Ingress
      15. EOF

      policy-group.network.openshift.io/ingress: “” is the preferred namespace selector label for OpenShift SDN. You can use the network.openshift.io/policy-group: ingress namespace selector label, but this is a legacy label.

    2. A policy named allow-from-openshift-monitoring:

      1. $ cat << EOF| oc create -f -
      2. apiVersion: networking.k8s.io/v1
      3. kind: NetworkPolicy
      4. metadata:
      5.   name: allow-from-openshift-monitoring
      6. spec:
      7.   ingress:
      8.   - from:
      9.     - namespaceSelector:
      10.         matchLabels:
      11.           network.openshift.io/policy-group: monitoring
      12.   podSelector: {}
      13.   policyTypes:
      14.   - Ingress
      15. EOF

    3. A policy named allow-same-namespace:

      1. $ cat << EOF| oc create -f -
      2. kind: NetworkPolicy
      3. apiVersion: networking.k8s.io/v1
      4. metadata:
      5.   name: allow-same-namespace
      6. spec:
      7.   podSelector:
      8.   ingress:
      9.   - from:
      10.     - podSelector: {}
      11. EOF

    4. A policy named allow-from-kube-apiserver-operator:

      1. $ cat << EOF| oc create -f -
      2. apiVersion: networking.k8s.io/v1
      3. kind: NetworkPolicy
      4. metadata:
      5.   name: allow-from-kube-apiserver-operator
      6. spec:
      7.   ingress:
      8.   - from:
      9.     - namespaceSelector:
      10.         matchLabels:
      11.           kubernetes.io/metadata.name: openshift-kube-apiserver-operator
      12.       podSelector:
      13.         matchLabels:
      14.           app: kube-apiserver-operator
      15.   policyTypes:
      16.   - Ingress
      17. EOF

      For more details, see New kube-apiserver-operator webhook controller validating health of webhook.

  2. Optional: To confirm that the network policies exist in your current project, enter the following command:

    1. $ oc describe networkpolicy

    Example output

    ``` Name: allow-from-openshift-ingress Namespace: example1 Created on: 2020-06-09 00:28:17 -0400 EDT Labels: Annotations: Spec: PodSelector: (Allowing the specific traffic to all pods in this namespace) Allowing ingress traffic:

    1. To Port: <any> (traffic allowed to all ports)
    2. From:
    3.   NamespaceSelector: network.openshift.io/policy-group: ingress

    Not affecting egress traffic Policy Types: Ingress

  1. Name:         allow-from-openshift-monitoring
  2. Namespace:    example1
  3. Created on:   2020-06-09 00:29:57 -0400 EDT
  4. Labels:       <none>
  5. Annotations:  <none>
  6. Spec:
  7.   PodSelector:     <none> (Allowing the specific traffic to all pods in this namespace)
  8.   Allowing ingress traffic:
  9.     To Port: <any> (traffic allowed to all ports)
  10.     From:
  11.       NamespaceSelector: network.openshift.io/policy-group: monitoring
  12.   Not affecting egress traffic
  13.   Policy Types: Ingress
  14. ```

Creating default network policies for a new project

As a cluster administrator, you can modify the new project template to automatically include NetworkPolicy objects when you create a new project.

Modifying the template for new projects

As a cluster administrator, you can modify the default project template so that new projects are created using your custom requirements.

To create your own custom project template:

Procedure

  1. Log in as a user with cluster-admin privileges.

  2. Generate the default project template:

    1. $ oc adm create-bootstrap-project-template -o yaml > template.yaml

  3. Use a text editor to modify the generated template.yaml file by adding objects or modifying existing objects.

  4. The project template must be created in the openshift-config namespace. Load your modified template:

    1. $ oc create -f template.yaml -n openshift-config

  5. Edit the project configuration resource using the web console or CLI.

    • Using the web console:

      1. Navigate to the Administration → Cluster Settings page.

      2. Click Configuration to view all configuration resources.

      3. Find the entry for Project and click Edit YAML.

    • Using the CLI:

      1. Edit the project.config.openshift.io/cluster resource:

        1. $ oc edit project.config.openshift.io/cluster

  1. Update the spec section to include the projectRequestTemplate and name parameters, and set the name of your uploaded project template. The default name is project-request.

    Project configuration resource with custom project template

    1. apiVersion: config.openshift.io/v1
    2. kind: Project
    3. metadata:
    4.   ...
    5. spec:
    6.   projectRequestTemplate:
    7.     name: <template_name>

  2. After you save your changes, create a new project to verify that your changes were successfully applied.

Adding network policies to the new project template

As a cluster administrator, you can add network policies to the default template for new projects. OKD will automatically create all the NetworkPolicy objects specified in the template in the project.

Prerequisites

  • Your cluster uses a default CNI network plugin that supports NetworkPolicy objects, such as the OpenShift SDN network plugin with mode: NetworkPolicy set. This mode is the default for OpenShift SDN.

  • You installed the OpenShift CLI (oc).

  • You must log in to the cluster with a user with cluster-admin privileges.

  • You must have created a custom default project template for new projects.

Procedure

  1. Edit the default template for a new project by running the following command:

    1. $ oc edit template <project_template> -n openshift-config

    Replace <project_template> with the name of the default template that you configured for your cluster. The default template name is project-request.

  2. In the template, add each NetworkPolicy object as an element to the objects parameter. The objects parameter accepts a collection of one or more objects.

    In the following example, the objects parameter collection includes several NetworkPolicy objects.

    1. objects:
    2. - apiVersion: networking.k8s.io/v1
    3.   kind: NetworkPolicy
    4.   metadata:
    5.     name: allow-from-same-namespace
    6.   spec:
    7.     podSelector: {}
    8.     ingress:
    9.     - from:
    10.       - podSelector: {}
    11. - apiVersion: networking.k8s.io/v1
    12.   kind: NetworkPolicy
    13.   metadata:
    14.     name: allow-from-openshift-ingress
    15.   spec:
    16.     ingress:
    17.     - from:
    18.       - namespaceSelector:
    19.           matchLabels:
    20.             network.openshift.io/policy-group: ingress
    21.     podSelector: {}
    22.     policyTypes:
    23.     - Ingress
    24. - apiVersion: networking.k8s.io/v1
    25.   kind: NetworkPolicy
    26.   metadata:
    27.     name: allow-from-kube-apiserver-operator
    28.   spec:
    29.     ingress:
    30.     - from:
    31.       - namespaceSelector:
    32.           matchLabels:
    33.             kubernetes.io/metadata.name: openshift-kube-apiserver-operator
    34.         podSelector:
    35.           matchLabels:
    36.             app: kube-apiserver-operator
    37.     policyTypes:
    38.     - Ingress
    39. ...

  3. Optional: Create a new project to confirm that your network policy objects are created successfully by running the following commands:

    1. Create a new project:

      1. $ oc new-project <project> (1)
      1Replace <project> with the name for the project you are creating.

    2. Confirm that the network policy objects in the new project template exist in the new project:

      1. $ oc get networkpolicy
      2. NAME                           POD-SELECTOR   AGE
      3. allow-from-openshift-ingress   <none>         7s
      4. allow-from-same-namespace      <none>         7s

Optimizing routing

The OKD HAProxy router can be scaled or configured to optimize performance.

Baseline Ingress Controller (router) performance

The OKD Ingress Controller, or router, is the ingress point for ingress traffic for applications and services that are configured using routes and ingresses.

When evaluating a single HAProxy router performance in terms of HTTP requests handled per second, the performance varies depending on many factors. In particular:

  • HTTP keep-alive/close mode

  • Route type

  • TLS session resumption client support

  • Number of concurrent connections per target route

  • Number of target routes

  • Back end server page size

  • Underlying infrastructure (network/SDN solution, CPU, and so on)

While performance in your specific environment will vary, Red Hat lab tests on a public cloud instance of size 4 vCPU/16GB RAM. A single HAProxy router handling 100 routes terminated by backends serving 1kB static pages is able to handle the following number of transactions per second.

In HTTP keep-alive mode scenarios:

EncryptionLoadBalancerServiceHostNetwork

none

21515

29622

edge

16743

22913

passthrough

36786

53295

re-encrypt

21583

25198

In HTTP close (no keep-alive) scenarios:

EncryptionLoadBalancerServiceHostNetwork

none

5719

8273

edge

2729

4069

passthrough

4121

5344

re-encrypt

2320

2941

The default Ingress Controller configuration was used with the spec.tuningOptions.threadCount field set to 4. Two different endpoint publishing strategies were tested: Load Balancer Service and Host Network. TLS session resumption was used for encrypted routes. With HTTP keep-alive, a single HAProxy router is capable of saturating a 1 Gbit NIC at page sizes as small as 8 kB.

When running on bare metal with modern processors, you can expect roughly twice the performance of the public cloud instance above. This overhead is introduced by the virtualization layer in place on public clouds and holds mostly true for private cloud-based virtualization as well. The following table is a guide to how many applications to use behind the router:

Number of applicationsApplication type

5-10

static file/web server or caching proxy

100-1000

applications generating dynamic content

In general, HAProxy can support routes for up to 1000 applications, depending on the technology in use. Ingress Controller performance might be limited by the capabilities and performance of the applications behind it, such as language or static versus dynamic content.

Ingress, or router, sharding should be used to serve more routes towards applications and help horizontally scale the routing tier.

Configuring Ingress Controller liveness, readiness, and startup probes

Cluster administrators can configure the timeout values for the kubelet’s liveness, readiness, and startup probes for router deployments that are managed by the OKD Ingress Controller (router). The liveness and readiness probes of the router use the default timeout value of 1 second, which is too brief when networking or runtime performance is severely degraded. Probe timeouts can cause unwanted router restarts that interrupt application connections. The ability to set larger timeout values can reduce the risk of unnecessary and unwanted restarts.

You can update the timeoutSeconds value on the livenessProbe, readinessProbe, and startupProbe parameters of the router container.

ParameterDescription

livenessProbe

The livenessProbe reports to the kubelet whether a pod is dead and needs to be restarted.

readinessProbe

The readinessProbe reports whether a pod is healthy or unhealthy. When the readiness probe reports an unhealthy pod, then the kubelet marks the pod as not ready to accept traffic. Subsequently, the endpoints for that pod are marked as not ready, and this status propagates to the kube-proxy. On cloud platforms with a configured load balancer, the kube-proxy communicates to the cloud load-balancer not to send traffic to the node with that pod.

startupProbe

The startupProbe gives the router pod up to 2 minutes to initialize before the kubelet begins sending the router liveness and readiness probes. This initialization time can prevent routers with many routes or endpoints from prematurely restarting.

The timeout configuration option is an advanced tuning technique that can be used to work around issues. However, these issues should eventually be diagnosed and possibly a support case or Jira issue opened for any issues that causes probes to time out.

The following example demonstrates how you can directly patch the default router deployment to set a 5-second timeout for the liveness and readiness probes:

  1. $ oc -n openshift-ingress patch deploy/router-default --type=strategic --patch='{"spec":{"template":{"spec":{"containers":[{"name":"router","livenessProbe":{"timeoutSeconds":5},"readinessProbe":{"timeoutSeconds":5}}]}}}}'

Verification

  1. $ oc -n openshift-ingress describe deploy/router-default | grep -e Liveness: -e Readiness:
  2.     Liveness:   http-get http://:1936/healthz delay=0s timeout=5s period=10s #success=1 #failure=3
  3.     Readiness:  http-get http://:1936/healthz/ready delay=0s timeout=5s period=10s #success=1 #failure=3

Configuring HAProxy reload interval

When you update a route or an endpoint associated with a route, OKD router updates the configuration for HAProxy. Then, HAProxy reloads the updated configuration for those changes to take effect. When HAProxy reloads, it generates a new process that handles new connections using the updated configuration.

HAProxy keeps the old process running to handle existing connections until those connections are all closed. When old processes have long-lived connections, these processes can accumulate and consume resources.

The default minimum HAProxy reload interval is five seconds. You can configure an Ingress Controller using its spec.tuningOptions.reloadInterval field to set a longer minimum reload interval.

Setting a large value for the minimum HAProxy reload interval can cause latency in observing updates to routes and their endpoints. To lessen the risk, avoid setting a value larger than the tolerable latency for updates.

Procedure

  • Change the minimum HAProxy reload interval of the default Ingress Controller to 15 seconds by running the following command:

    1. $ oc -n openshift-ingress-operator patch ingresscontrollers/default --type=merge --patch='{"spec":{"tuningOptions":{"reloadInterval":"15s"}}}'

Postinstallation OpenStack network configuration

You can configure some aspects of an OKD on OpenStack cluster after installation.

Configuring application access with floating IP addresses

After you install OKD, configure OpenStack to allow application network traffic.

You do not need to perform this procedure if you provided values for platform.openstack.apiFloatingIP and platform.openstack.ingressFloatingIP in the install-config.yaml file, or os_api_fip and os_ingress_fip in the inventory.yaml playbook, during installation. The floating IP addresses are already set.

Prerequisites

  • OKD cluster must be installed

  • Floating IP addresses are enabled as described in the OKD on OpenStack installation documentation.

Procedure

After you install the OKD cluster, attach a floating IP address to the ingress port:

  1. Show the port:

    1. $ openstack port show <cluster_name>-<cluster_ID>-ingress-port

  2. Attach the port to the IP address:

    1. $ openstack floating ip set --port <ingress_port_ID> <apps_FIP>

  3. Add a wildcard A record for *apps. to your DNS file:

    1. *.apps.<cluster_name>.<base_domain>  IN  A  <apps_FIP>

If you do not control the DNS server but want to enable application access for non-production purposes, you can add these hostnames to /etc/hosts:

  1. <apps_FIP> console-openshift-console.apps.<cluster name>.<base domain>
  2. <apps_FIP> integrated-oauth-server-openshift-authentication.apps.<cluster name>.<base domain>
  3. <apps_FIP> oauth-openshift.apps.<cluster name>.<base domain>
  4. <apps_FIP> prometheus-k8s-openshift-monitoring.apps.<cluster name>.<base domain>
  5. <apps_FIP> <app name>.apps.<cluster name>.<base domain>

Kuryr ports pools

A Kuryr ports pool maintains a number of ports on standby for pod creation.

Keeping ports on standby minimizes pod creation time. Without ports pools, Kuryr must explicitly request port creation or deletion whenever a pod is created or deleted.

The Neutron ports that Kuryr uses are created in subnets that are tied to namespaces. These pod ports are also added as subports to the primary port of OKD cluster nodes.

Because Kuryr keeps each namespace in a separate subnet, a separate ports pool is maintained for each namespace-worker pair.

Prior to installing a cluster, you can set the following parameters in the cluster-network-03-config.yml manifest file to configure ports pool behavior:

  • The enablePortPoolsPrepopulation parameter controls pool prepopulation, which forces Kuryr to add Neutron ports to the pools when the first pod that is configured to use the dedicated network for pods is created in a namespace. The default value is false.

  • The poolMinPorts parameter is the minimum number of free ports that are kept in the pool. The default value is 1.

  • The poolMaxPorts parameter is the maximum number of free ports that are kept in the pool. A value of 0 disables that upper bound. This is the default setting.

    If your OpenStack port quota is low, or you have a limited number of IP addresses on the pod network, consider setting this option to ensure that unneeded ports are deleted.

  • The poolBatchPorts parameter defines the maximum number of Neutron ports that can be created at once. The default value is 3.

Adjusting Kuryr ports pool settings in active deployments on OpenStack

You can use a custom resource (CR) to configure how Kuryr manages OpenStack Neutron ports to control the speed and efficiency of pod creation on a deployed cluster.

Procedure

  1. From a command line, open the Cluster Network Operator (CNO) CR for editing:

    1. $ oc edit networks.operator.openshift.io cluster

  2. Edit the settings to meet your requirements. The following file is provided as an example:

    1. apiVersion: operator.openshift.io/v1
    2. kind: Network
    3. metadata:
    4.   name: cluster
    5. spec:
    6.   clusterNetwork:
    7.   - cidr: 10.128.0.0/14
    8.     hostPrefix: 23
    9.   serviceNetwork:
    10.   - 172.30.0.0/16
    11.   defaultNetwork:
    12.     type: Kuryr
    13.     kuryrConfig:
    14.       enablePortPoolsPrepopulation: false (1)
    15.       poolMinPorts: 1 (2)
    16.       poolBatchPorts: 3 (3)
    17.       poolMaxPorts: 5 (4)
    1Set enablePortPoolsPrepopulation to true to make Kuryr create Neutron ports when the first pod that is configured to use the dedicated network for pods is created in a namespace. This setting raises the Neutron ports quota but can reduce the time that is required to spawn pods. The default value is false.
    2Kuryr creates new ports for a pool if the number of free ports in that pool is lower than the value of poolMinPorts. The default value is 1.
    3poolBatchPorts controls the number of new ports that are created if the number of free ports is lower than the value of poolMinPorts. The default value is 3.
    4If the number of free ports in a pool is higher than the value of poolMaxPorts, Kuryr deletes them until the number matches that value. Setting the value to 0 disables this upper bound, preventing pools from shrinking. The default value is 0.

  3. Save your changes and quit the text editor to commit your changes.

Modifying these options on a running cluster forces the kuryr-controller and kuryr-cni pods to restart. As a result, the creation of new pods and services will be delayed.

Enabling OVS hardware offloading

For clusters that run on OpenStack, you can enable Open vSwitch (OVS) hardware offloading.

OVS is a multi-layer virtual switch that enables large-scale, multi-server network virtualization.

Prerequisites

  • You installed a cluster on OpenStack that is configured for single-root input/output virtualization (SR-IOV).

  • You installed the SR-IOV Network Operator on your cluster.

  • You created two hw-offload type virtual function (VF) interfaces on your cluster.

Procedure

  1. Create an SriovNetworkNodePolicy policy for the two hw-offload type VF interfaces that are on your cluster:

    The first virtual function interface

    1. apiVersion: sriovnetwork.openshift.io/v1
    2. kind: SriovNetworkNodePolicy (1)
    3. metadata:
    4.   name: "hwoffload9"
    5.   namespace: openshift-sriov-network-operator
    6. spec:
    7.   deviceType: netdevice
    8.   isRdma: true
    9.   nicSelector:
    10.     pfNames: (2)
    11.     - ens6
    12.   nodeSelector:
    13.     feature.node.kubernetes.io/network-sriov.capable: 'true'
    14.   numVfs: 1
    15.   priority: 99
    16.   resourceName: "hwoffload9"
    1Insert the SriovNetworkNodePolicy value here.
    2Both interfaces must include physical function (PF) names.

    The second virtual function interface

    1. apiVersion: sriovnetwork.openshift.io/v1
    2. kind: SriovNetworkNodePolicy (1)
    3. metadata:
    4.   name: "hwoffload10"
    5.   namespace: openshift-sriov-network-operator
    6. spec:
    7.   deviceType: netdevice
    8.   isRdma: true
    9.   nicSelector:
    10.     pfNames: (2)
    11.     - ens5
    12.   nodeSelector:
    13.     feature.node.kubernetes.io/network-sriov.capable: 'true'
    14.   numVfs: 1
    15.   priority: 99
    16.   resourceName: "hwoffload10"
    1Insert the SriovNetworkNodePolicy value here.
    2Both interfaces must include physical function (PF) names.

  2. Create NetworkAttachmentDefinition resources for the two interfaces:

    A NetworkAttachmentDefinition resource for the first interface

    1. apiVersion: k8s.cni.cncf.io/v1
    2. kind: NetworkAttachmentDefinition
    3. metadata:
    4.   annotations:
    5.     k8s.v1.cni.cncf.io/resourceName: openshift.io/hwoffload9
    6.   name: hwoffload9
    7.   namespace: default
    8. spec:
    9.     config: '{ "cniVersion":"0.3.1", "name":"hwoffload9","type":"host-device","device":"ens6"
    10.     }'

    A NetworkAttachmentDefinition resource for the second interface

    1. apiVersion: k8s.cni.cncf.io/v1
    2. kind: NetworkAttachmentDefinition
    3. metadata:
    4.   annotations:
    5.     k8s.v1.cni.cncf.io/resourceName: openshift.io/hwoffload10
    6.   name: hwoffload10
    7.   namespace: default
    8. spec:
    9.     config: '{ "cniVersion":"0.3.1", "name":"hwoffload10","type":"host-device","device":"ens5"
    10.     }'

  3. Use the interfaces that you created with a pod. For example:

    A pod that uses the two OVS offload interfaces

    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4.   name: dpdk-testpmd
    5.   namespace: default
    6.   annotations:
    7.     irq-load-balancing.crio.io: disable
    8.     cpu-quota.crio.io: disable
    9.     k8s.v1.cni.cncf.io/resourceName: openshift.io/hwoffload9
    10.     k8s.v1.cni.cncf.io/resourceName: openshift.io/hwoffload10
    11. spec:
    12.   restartPolicy: Never
    13.   containers:
    14.   - name: dpdk-testpmd
    15.     image: quay.io/krister/centos8_nfv-container-dpdk-testpmd:latest

Attaching an OVS hardware offloading network

You can attach an Open vSwitch (OVS) hardware offloading network to your cluster.

Prerequisites

  • Your cluster is installed and running.

  • You provisioned an OVS hardware offloading network on OpenStack to use with your cluster.

Procedure

  1. Create a file named network.yaml from the following template:

    1. spec:
    2.   additionalNetworks:
    3.   - name: hwoffload1
    4.     namespace: cnf
    5.     rawCNIConfig: '{ "cniVersion": "0.3.1", "name": "hwoffload1", "type": "host-device","pciBusId": "0000:00:05.0", "ipam": {}}' (1)
    6.     type: Raw

    where:

    pciBusId

    Specifies the device that is connected to the offloading network. If you do not have it, you can find this value by running the following command:

    1. $ oc describe SriovNetworkNodeState -n openshift-sriov-network-operator

  2. From a command line, enter the following command to patch your cluster with the file:

    1. $ oc apply -f network.yaml

Enabling IPv6 connectivity to pods on OpenStack

To enable IPv6 connectivity between pods that have additional networks that are on different nodes, disable port security for the IPv6 port of the server. Disabling port security obviates the need to create allowed address pairs for each IPv6 address that is assigned to pods and enables traffic on the security group.

Only the following IPv6 additional network configurations are supported:

  • SLAAC and host-device

  • SLAAC and MACVLAN

  • DHCP stateless and host-device

  • DHCP stateless and MACVLAN

Procedure

  • On a command line, enter the following command:

    1. $ openstack port set --no-security-group --disable-port-security <compute_ipv6_port>
    This command removes security groups from the port and disables port security. Traffic restrictions are removed entirely from the port.

where:

<compute_ipv6_port>

Specifies the IPv6 port of the compute server.

Adding IPv6 connectivity to pods on OpenStack

After you enable IPv6 connectivity in pods, add connectivity to them by using a Container Network Interface (CNI) configuration.

Procedure

  1. To edit the Cluster Network Operator (CNO), enter the following command:

    1. $ oc edit networks.operator.openshift.io cluster

  2. Specify your CNI configuration under the spec field. For example, the following configuration uses a SLAAC address mode with MACVLAN:

    1. ...
    2. spec:
    3.   additionalNetworks:
    4.   - name: ipv6
    5.     namespace: ipv6 (1)
    6.     rawCNIConfig: '{ "cniVersion": "0.3.1", "name": "ipv6", "type": "macvlan", "master": "ens4"}' (2)
    7.     type: Raw
    1Be sure to create pods in the same namespace.
    2The interface in the network attachment “master” field can differ from “ens4” when more networks are configured or when a different kernel driver is used.

    If you are using stateful address mode, include the IP Address Management (IPAM) in the CNI configuration.

    DHCPv6 is not supported by Multus.

  3. Save your changes and quit the text editor to commit your changes.

Verification

  • On a command line, enter the following command:

    1. $ oc get network-attachment-definitions -A

    Example output

    1. NAMESPACE       NAME            AGE
    2. ipv6            ipv6            21h

You can now create pods that have secondary IPv6 connections.

Additional resources

Create pods that have IPv6 connectivity on OpenStack

After you enable IPv6 connectivty for pods and add it to them, create pods that have secondary IPv6 connections.

Procedure

  1. Define pods that use your IPv6 namespace and the annotation k8s.v1.cni.cncf.io/networks: <additional_network_name>, where <additional_network_name is the name of the additional network. For example, as part of a Deployment object:

    1. apiVersion: apps/v1
    2. kind: Deployment
    3. metadata:
    4.   name: hello-openshift
    5.   namespace: ipv6
    6. spec:
    7.   affinity:
    8.     podAntiAffinity:
    9.       requiredDuringSchedulingIgnoredDuringExecution:
    10.          - labelSelector:
    11.             matchExpressions:
    12.             - key: app
    13.               operator: In
    14.               values:
    15.               - hello-openshift
    16.   replicas: 2
    17.   selector:
    18.     matchLabels:
    19.       app: hello-openshift
    20.   template:
    21.     metadata:
    22.       labels:
    23.         app: hello-openshift
    24.       annotations:
    25.         k8s.v1.cni.cncf.io/networks: ipv6
    26.     spec:
    27.       securityContext:
    28.         runAsNonRoot: true
    29.         seccompProfile:
    30.           type: RuntimeDefault
    31.       containers:
    32.       - name: hello-openshift
    33.         securityContext:
    34.           allowPrivilegeEscalation: false
    35.           capabilities:
    36.             drop:
    37.             - ALL
    38.         image: quay.io/openshift/origin-hello-openshift
    39.         ports:
    40.         - containerPort: 8080

  2. Create the pod. For example, on a command line, enter the following command:

    1. $ oc create -f <ipv6_enabled_resource>

where:

<ipv6_enabled_resource>

Specifies the file that contains your resource definition.