Configuring an additional network

As a cluster administrator, you can configure an additional network for your cluster. The following network types are supported:

• Bridge

• Host device

• VLAN

• IPVLAN

• MACVLAN

• TAP

• OVN-Kubernetes

Approaches to managing an additional network

You can manage the life cycle of an additional network by two approaches. Each approach is mutually exclusive and you can only use one approach for managing an additional network at a time. For either approach, the additional network is managed by a Container Network Interface (CNI) plugin that you configure.

For an additional network, IP addresses are provisioned through an IP Address Management (IPAM) CNI plugin that you configure as part of the additional network. The IPAM plugin supports a variety of IP address assignment approaches including DHCP and static assignment.

• Modify the Cluster Network Operator (CNO) configuration: The CNO automatically creates and manages the NetworkAttachmentDefinition object. In addition to managing the object lifecycle the CNO ensures a DHCP is available for an additional network that uses a DHCP assigned IP address.

• Applying a YAML manifest: You can manage the additional network directly by creating an NetworkAttachmentDefinition object. This approach allows for the chaining of CNI plugins.

$ openstack subnet set —dns-nameserver 0.0.0.0 <subnet_id>

Configuration for an additional network attachment

An additional network is configured by using the NetworkAttachmentDefinition API in the k8s.cni.cncf.io API group.

The configuration for the API is described in the following table:

Configuration of an additional network through the Cluster Network Operator

The configuration for an additional network attachment is specified as part of the Cluster Network Operator (CNO) configuration.

The following YAML describes the configuration parameters for managing an additional network with the CNO:

Cluster Network Operator configuration

apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  name: cluster
spec:
  # ...
  additionalNetworks: (1)
  - name: <name> (2)
    namespace: <namespace> (3)
    rawCNIConfig: |- (4)
      {
        ...
      }
    type: Raw

Configuration of an additional network from a YAML manifest

The configuration for an additional network is specified from a YAML configuration file, such as in the following example:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: <name> (1)
spec:
  config: |- (2)
    {
      ...
    }

Configurations for additional network types

The specific configuration fields for additional networks is described in the following sections.

Configuration for a bridge additional network

The following object describes the configuration parameters for the bridge CNI plugin:

$  bridge vlan add vid VLAN_ID dev DEV

bridge configuration example

The following example configures an additional network named bridge-net:

{
  "cniVersion": "0.3.1",
  "name": "bridge-net",
  "type": "bridge",
  "isGateway": true,
  "vlan": 2,
  "ipam": {
    "type": "dhcp"
    }
}

Configuration for a host device additional network

The following object describes the configuration parameters for the host-device CNI plugin:

host-device configuration example

The following example configures an additional network named hostdev-net:

{
  "cniVersion": "0.3.1",
  "name": "hostdev-net",
  "type": "host-device",
  "device": "eth1"
}

Configuration for an VLAN additional network

The following object describes the configuration parameters for the VLAN CNI plugin:

vlan configuration example

The following example configures an additional network named vlan-net:

{
  "name": "vlan-net",
  "cniVersion": "0.3.1",
  "type": "vlan",
  "master": "eth0",
  "mtu": 1500,
  "vlanId": 5,
  "linkInContainer": false,
  "ipam": {
      "type": "host-local",
      "subnet": "10.1.1.0/24"
  },
  "dns": {
      "nameservers": [ "10.1.1.1", "8.8.8.8" ]
  }
}

Configuration for an IPVLAN additional network

The following object describes the configuration parameters for the IPVLAN CNI plugin:

ipvlan configuration example

The following example configures an additional network named ipvlan-net:

{
  "cniVersion": "0.3.1",
  "name": "ipvlan-net",
  "type": "ipvlan",
  "master": "eth1",
  "linkInContainer": false,
  "mode": "l3",
  "ipam": {
    "type": "static",
    "addresses": [
       {
         "address": "192.168.10.10/24"
       }
    ]
  }
}

Configuration for a MACVLAN additional network

The following object describes the configuration parameters for the macvlan CNI plugin:

macvlan configuration example

The following example configures an additional network named macvlan-net:

{
  "cniVersion": "0.3.1",
  "name": "macvlan-net",
  "type": "macvlan",
  "master": "eth1",
  "linkInContainer": false,
  "mode": "bridge",
  "ipam": {
    "type": "dhcp"
    }
}

Configuration for a TAP additional network

The following object describes the configuration parameters for the TAP CNI plugin:

Tap configuration example

The following example configures an additional network named mynet:

{
 "name": "mynet",
 "cniVersion": "0.3.1",
 "type": "tap",
 "mac": "00:11:22:33:44:55",
 "mtu": 1500,
 "selinuxcontext": "system_u:system_r:container_t:s0",
 "multiQueue": true,
 "owner": 0,
 "group": 0
 "bridge": "br1"
}

Setting SELinux boolean for the TAP CNI plugin

To create the tap device with the container_t SELinux context, enable the container_use_devices boolean on the host by using the Machine Config Operator (MCO).

Prerequisites

• You have installed the OpenShift CLI (oc).

Procedure

1. Create a new YAML file named, such as setsebool-container-use-devices.yaml, with the following details:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 99-worker-setsebool
   spec:
     config:
       ignition:
         version: 3.2.0
       systemd:
         units:
         - enabled: true
           name: setsebool.service
           contents: |
             [Unit]
             Description=Set SELinux boolean for the TAP CNI plugin
             Before=kubelet.service
             [Service]
             Type=oneshot
             ExecStart=/usr/sbin/setsebool container_use_devices=on
             RemainAfterExit=true
             [Install]
             WantedBy=multi-user.target graphical.target

2. Create the new MachineConfig object by running the following command:

   $ oc apply -f setsebool-container-use-devices.yaml

   Applying any changes to the MachineConfig object causes all affected nodes to gracefully reboot after the change is applied. This update can take some time to be applied.

3. Verify the change is applied by running the following command:

   $ oc get machineconfigpools

   Expected output

   NAME        CONFIG                                                UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master      rendered-master-e5e0c8e8be9194e7c5a882e047379cfa      True      False      False      3              3                   3                     0                      7d2h
   worker      rendered-worker-d6c9ca107fba6cd76cdcbfcedcafa0f2      True      False      False      3              3                   3                     0                      7d

   All nodes should be in the updated and ready state.

Additional resources

• For more information about enabling an SELinux boolean on a node, see Setting SELinux booleans

Configuration for an OVN-Kubernetes additional network

The Red Hat OpenShift Networking OVN-Kubernetes network plugin allows the configuration of secondary network interfaces for pods. To configure secondary network interfaces, you must define the configurations in the NetworkAttachmentDefinition custom resource (CR).

You can configure an OVN-Kubernetes additional network in either layer 2 or localnet topologies.

• A layer 2 topology supports east-west cluster traffic, but does not allow access to the underlying physical network.

• A localnet topology allows connections to the physical network, but requires additional configuration of the underlying Open vSwitch (OVS) bridge on cluster nodes.

The following sections provide example configurations for each of the topologies that OVN-Kubernetes currently allows for secondary networks.

Supported platforms for OVN-Kubernetes additional network

You can use an OVN-Kubernetes additional network with the following supported platforms:

• Bare metal

• IBM Power®

• IBM Z®

• IBM® LinuxONE

• VMware vSphere

• OpenStack

OVN-Kubernetes network plugin JSON configuration table

The following table describes the configuration parameters for the OVN-Kubernetes CNI network plugin:

Compatibility with multi-network policy

The multi-network policy API, which is provided by the MultiNetworkPolicy custom resource definition (CRD) in the k8s.cni.cncf.io API group, is compatible with an OVN-Kubernetes secondary network. When defining a network policy, the network policy rules that can be used depend on whether the OVN-Kubernetes secondary network defines the subnets field. Refer to the following table for details:

For example, the following multi-network policy is valid only if the subnets field is defined in the additional network CNI configuration for the additional network named blue2:

Example multi-network policy that uses a pod selector

apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  name: allow-same-namespace
  annotations:
    k8s.v1.cni.cncf.io/policy-for: blue2
spec:
  podSelector:
  ingress:
  - from:
    - podSelector: {}

The following example uses the ipBlock network policy selector, which is always valid for an OVN-Kubernetes additional network:

Example multi-network policy that uses an IP block selector

apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  name:  ingress-ipblock
  annotations:
    k8s.v1.cni.cncf.io/policy-for: default/flatl2net
spec:
  podSelector:
    matchLabels:
      name: access-control
  policyTypes:
  - Ingress
  ingress:
  - from:
    - ipBlock:
        cidr: 10.200.0.0/30

Configuration for a layer 2 switched topology

The switched (layer 2) topology networks interconnect the workloads through a cluster-wide logical switch. This configuration can be used for IPv6 and dual-stack deployments.

The following JSON example configures a switched secondary network:

{
  "cniVersion": "0.3.1",
  "name": "l2-network",
  "type": "ovn-k8s-cni-overlay",
  "topology":"layer2",
  "subnets": "10.100.200.0/24",
  "mtu": 1300,
  "netAttachDefName": "ns1/l2-network",
  "excludeSubnets": "10.100.200.0/29"
}

Configuration for a localnet topology

The switched (localnet) topology interconnects the workloads through a cluster-wide logical switch to a physical network.

Prerequisites for configuring OVN-Kubernetes additional network

• The NMState Operator is installed. For more information, see About the Kubernetes NMState Operator.

Configuration for an OVN-Kubernetes additional network mapping

You must map an additional network to the OVN bridge to use it as an OVN-Kubernetes additional network. Bridge mappings allow network traffic to reach the physical network. A bridge mapping associates a physical network name, also known as an interface label, to a bridge created with Open vSwitch (OVS).

You can create an NodeNetworkConfigurationPolicy object, part of the nmstate.io/v1 API group, to declaratively create the mapping. This API is provided by the NMState Operator. By using this API you can apply the bridge mapping to nodes that match your specified nodeSelector expression, such as node-role.kubernetes.io/worker: ''.

When attaching an additional network, you can either use the existing br-ex bridge or create a new bridge. Which approach to use depends on your specific network infrastructure.

• If your nodes include only a single network interface, you must use the existing bridge. This network interface is owned and managed by OVN-Kubernetes and you must not remove it from the br-ex bridge or alter the interface configuration. If you remove or alter the network interface, your cluster network will stop working correctly.

• If your nodes include several network interfaces, you can attach a different network interface to a new bridge, and use that for your additional network. This approach provides for traffic isolation from your primary cluster network.

The localnet1 network is mapped to the br-ex bridge in the following example:

Example mapping for sharing a bridge

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: mapping (1)
spec:
  nodeSelector:
    node-role.kubernetes.io/worker: '' (2)
  desiredState:
    ovn:
      bridge-mappings:
      - localnet: localnet1 (3)
        bridge: br-ex (4)
        state: present (5)

In the following example, the localnet2 network interface is attached to the ovs-br1 bridge. Through this attachment, the network interface is available to the OVN-Kubernetes network plugin as an additional network.

Example mapping for nodes with multiple interfaces

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: ovs-br1-multiple-networks (1)
spec:
  nodeSelector:
    node-role.kubernetes.io/worker: '' (2)
  desiredState:
    interfaces:
    - name: ovs-br1 (3)
      description: |-
        A dedicated OVS bridge with eth1 as a port
        allowing all VLANs and untagged traffic
      type: ovs-bridge
      state: up
      bridge:
        options:
          stp: true
        port:
        - name: eth1 (4)
    ovn:
      bridge-mappings:
      - localnet: localnet2 (5)
        bridge: ovs-br1 (6)
        state: present (7)

This declarative approach is recommended because the NMState Operator applies additional network configuration to all nodes specified by the node selector automatically and transparently.

The following JSON example configures a localnet secondary network:

{
  "cniVersion": "0.3.1",
  "name": "ns1-localnet-network",
  "type": "ovn-k8s-cni-overlay",
  "topology":"localnet",
  "subnets": "202.10.130.112/28",
  "vlanID": 33,
  "mtu": 1500,
  "netAttachDefName": "ns1/localnet-network"
  "excludeSubnets": "10.100.200.0/29"
}

Configuring pods for additional networks

You must specify the secondary network attachments through the k8s.v1.cni.cncf.io/networks annotation.

The following example provisions a pod with two secondary attachments, one for each of the attachment configurations presented in this guide.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: l2-network
  name: tinypod
  namespace: ns1
spec:
  containers:
  - args:
    - pause
    image: k8s.gcr.io/e2e-test-images/agnhost:2.36
    imagePullPolicy: IfNotPresent
    name: agnhost-container

Configuring pods with a static IP address

The following example provisions a pod with a static IP address.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
      {
        "name": "l2-network", (1)
        "mac": "02:03:04:05:06:07", (2)
        "interface": "myiface1", (3)
        "ips": [
          "192.0.2.20/24"
          ] (4)
      }
    ]'
  name: tinypod
  namespace: ns1
spec:
  containers:
  - args:
    - pause
    image: k8s.gcr.io/e2e-test-images/agnhost:2.36
    imagePullPolicy: IfNotPresent
    name: agnhost-container

Configuration of IP address assignment for an additional network

The IP address management (IPAM) Container Network Interface (CNI) plugin provides IP addresses for other CNI plugins.

You can use the following IP address assignment types:

• Static assignment.

• Dynamic assignment through a DHCP server. The DHCP server you specify must be reachable from the additional network.

• Dynamic assignment through the Whereabouts IPAM CNI plugin.

Static IP address assignment configuration

The following table describes the configuration for static IP address assignment:

The addresses array requires objects with the following fields:

Static IP address assignment configuration example

{
  "ipam": {
    "type": "static",
      "addresses": [
        {
          "address": "191.168.1.7/24"
        }
      ]
  }
}

Dynamic IP address (DHCP) assignment configuration

The following JSON describes the configuration for dynamic IP address address assignment with DHCP.

apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  name: cluster
spec:
  additionalNetworks:
  - name: dhcp-shim
    namespace: default
    type: Raw
    rawCNIConfig: |-
      {
        “name”: “dhcp-shim”,
        “cniVersion”: “0.3.1”,
        “type”: “bridge”,
        “ipam”: {
          “type”: “dhcp”
        }
      }
  # …

Dynamic IP address (DHCP) assignment configuration example

{
  "ipam": {
    "type": "dhcp"
  }
}

Dynamic IP address assignment configuration with Whereabouts

The Whereabouts CNI plugin allows the dynamic assignment of an IP address to an additional network without the use of a DHCP server.

The following table describes the configuration for dynamic IP address assignment with Whereabouts:

Dynamic IP address assignment configuration example that uses Whereabouts

{
  "ipam": {
    "type": "whereabouts",
    "range": "192.0.2.192/27",
    "exclude": [
       "192.0.2.192/30",
       "192.0.2.196/32"
    ]
  }
}

Creating a Whereabouts reconciler daemon set

The Whereabouts reconciler is responsible for managing dynamic IP address assignments for the pods within a cluster using the Whereabouts IP Address Management (IPAM) solution. It ensures that each pods gets a unique IP address from the specified IP address range. It also handles IP address releases when pods are deleted or scaled down.

The Whereabouts reconciler daemon set is automatically created when you configure an additional network through the Cluster Network Operator. It is not automatically created when you configure an additional network from a YAML manifest.

To trigger the deployment of the Whereabouts reconciler daemonset, you must manually create a whereabouts-shim network attachment by editing the Cluster Network Operator custom resource file.

Use the following procedure to deploy the Whereabouts reconciler daemonset.

Procedure

1. Edit the Network.operator.openshift.io custom resource (CR) by running the following command:

   $ oc edit network.operator.openshift.io cluster

2. Modify the additionalNetworks parameter in the CR to add the whereabouts-shim network attachment definition. For example:

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     additionalNetworks:
     - name: whereabouts-shim
       namespace: default
       rawCNIConfig: |-
         {
          "name": "whereabouts-shim",
          "cniVersion": "0.3.1",
          "type": "bridge",
          "ipam": {
            "type": "whereabouts"
          }
         }
       type: Raw

3. Save the file and exit the text editor.

4. Verify that the whereabouts-reconciler daemon set deployed successfully by running the following command:

   $ oc get all -n openshift-multus | grep whereabouts-reconciler

   Example output

   pod/whereabouts-reconciler-jnp6g 1/1 Running 0 6s
   pod/whereabouts-reconciler-k76gg 1/1 Running 0 6s
   pod/whereabouts-reconciler-k86t9 1/1 Running 0 6s
   pod/whereabouts-reconciler-p4sxw 1/1 Running 0 6s
   pod/whereabouts-reconciler-rvfdv 1/1 Running 0 6s
   pod/whereabouts-reconciler-svzw9 1/1 Running 0 6s
   daemonset.apps/whereabouts-reconciler 6 6 6 6 6 kubernetes.io/os=linux 6s

Creating a configuration for assignment of dual-stack IP addresses dynamically

Dual-stack IP address assignment can be configured with the ipRanges parameter for:

• IPv4 addresses

• IPv6 addresses

• multiple IP address assignment

Procedure

1. Set type to whereabouts.

2. Use ipRanges to allocate IP addresses as shown in the following example:

   cniVersion: operator.openshift.io/v1
   kind: Network
   =metadata:
     name: cluster
   spec:
     additionalNetworks:
     - name: whereabouts-shim
       namespace: default
       type: Raw
       rawCNIConfig: |-
         {
          "name": "whereabouts-dual-stack",
          "cniVersion": "0.3.1,
          "type": "bridge",
          "ipam": {
            "type": "whereabouts",
            "ipRanges": [
                     {"range": "192.168.10.0/24"},
                     {"range": "2001:db8::/64"}
                 ]
          }
         }

3. Attach network to a pod. For more information, see “Adding a pod to an additional network”.

4. Verify that all IP addresses are assigned.

5. Run the following command to ensure the IP addresses are assigned as metadata.

   $ oc exec -it mypod -- ip a

Additional resources

• Attaching a pod to an additional network

Creating an additional network attachment with the Cluster Network Operator

The Cluster Network Operator (CNO) manages additional network definitions. When you specify an additional network to create, the CNO creates the NetworkAttachmentDefinition object automatically.

Prerequisites

• Install the OpenShift CLI (oc).

• Log in as a user with cluster-admin privileges.

Procedure

1. Optional: Create the namespace for the additional networks:

   $ oc create namespace <namespace_name>

2. To edit the CNO configuration, enter the following command:

   $ oc edit networks.operator.openshift.io cluster

3. Modify the CR that you are creating by adding the configuration for the additional network that you are creating, as in the following example CR.

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     # ...
     additionalNetworks:
     - name: tertiary-net
       namespace: namespace2
       type: Raw
       rawCNIConfig: |-
         {
           "cniVersion": "0.3.1",
           "name": "tertiary-net",
           "type": "ipvlan",
           "master": "eth1",
           "mode": "l2",
           "ipam": {
             "type": "static",
             "addresses": [
               {
                 "address": "192.168.1.23/24"
               }
             ]
           }
         }

4. Save your changes and quit the text editor to commit your changes.

Verification

• Confirm that the CNO created the NetworkAttachmentDefinition object by running the following command. There might be a delay before the CNO creates the object.

  $ oc get network-attachment-definitions -n <namespace>

  where:

  <namespace>

  Specifies the namespace for the network attachment that you added to the CNO configuration.

  Example output

  NAME                 AGE
  test-network-1       14m

Creating an additional network attachment by applying a YAML manifest

Prerequisites

• Install the OpenShift CLI (oc).

• Log in as a user with cluster-admin privileges.

Procedure

1. Create a YAML file with your additional network configuration, such as in the following example:

   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     name: next-net
   spec:
     config: |-
       {
         "cniVersion": "0.3.1",
         "name": "work-network",
         "type": "host-device",
         "device": "eth1",
         "ipam": {
           "type": "dhcp"
         }
       }

2. To create the additional network, enter the following command:

   $ oc apply -f <file>.yaml

   where:

   <file>

   Specifies the name of the file contained the YAML manifest.

About configuring the master interface in the container network namespace

In OKD 4.14 and later, the ability to allow users to create a MAC-VLAN, IP-VLAN, and VLAN subinterface based on a master interface in a container namespace is now generally available.

This feature allows you to create the master interfaces as part of the pod network configuration in a separate network attachment definition. You can then base the VLAN, MACVLAN, or IPVLAN on this interface without requiring the knowledge of the network configuration of the node.

To ensure the use of a container namespace master interface, specify the linkInContainer and set the value to true in the VLAN, MACVLAN, or IPVLAN plugin configuration depending on the particular type of additional network.

Creating multiple VLANs on SR-IOV VFs

An example use case for utilizing this feature is to create multiple VLANs based on SR-IOV VFs. To do so, begin by creating an SR-IOV network and then define the network attachments for the VLAN interfaces.

The following example shows how to configure the setup illustrated in this diagram.

Figure 1. Creating VLANs

Prerequisites

• You installed the OpenShift CLI (oc).

• You have access to the cluster as a user with the cluster-admin role.

• You have installed the SR-IOV Network Operator.

Procedure

1. Create a dedicated container namespace where you want to deploy your pod by using the following command:

   $ oc new-project test-namespace

2. Create an SR-IOV node policy:

   1. Create an SriovNetworkNodePolicy object, and then save the YAML in the sriov-node-network-policy.yaml file:

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetworkNodePolicy
      metadata:
       name: sriovnic
       namespace: openshift-sriov-network-operator
      spec:
       deviceType: netdevice
       isRdma: false
       needVhostNet: true
       nicSelector:
         vendor: "15b3" (1)
         deviceID: "101b" (2)
         rootDevices: ["00:05.0"]
       numVfs: 10
       priority: 99
       resourceName: sriovnic
       nodeSelector:
          feature.node.kubernetes.io/network-sriov.capable: "true"

      The SR-IOV network node policy configuration example, with the setting deviceType: netdevice, is tailored specifically for Mellanox Network Interface Cards (NICs).

      1	The vendor hexadecimal code of the SR-IOV network device. The value 15b3 is associated with a Mellanox NIC.
      2	The device hexadecimal code of the SR-IOV network device.

   2. Apply the YAML by running the following command:

      $ oc apply -f sriov-node-network-policy.yaml

      Applying this might take some time due to the node requiring a reboot.

3. Create an SR-IOV network:

   1. Create the SriovNetwork custom resource (CR) for the additional SR-IOV network attachment as in the following example CR. Save the YAML as the file sriov-network-attachment.yaml:

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetwork
      metadata:
       name: sriov-network
       namespace: openshift-sriov-network-operator
      spec:
       networkNamespace: test-namespace
       resourceName: sriovnic
       spoofChk: "off"
       trust: "on"

   2. Apply the YAML by running the following command:

      $ oc apply -f sriov-network-attachment.yaml

4. Create the VLAN additional network:

   1. Using the following YAML example, create a file named ipvlan100-additional-network-configuration.yaml:

      apiVersion: k8s.cni.cncf.io/v1
      kind: NetworkAttachmentDefinition
      metadata:
        name: vlan-100
        namespace: test-namespace
      spec:
        config: |
          {
            "cniVersion": "0.4.0",
            "name": "vlan-100",
            "plugins": [
              {
                "type": "vlan",
                "master": "ext0", (1)
                "mtu": 1500,
                "vlanId": 100,
                "linkInContainer": true, (2)
                "ipam": {"type": "whereabouts", "ipRanges": [{"range": "1.1.1.0/24"}]}
              }
            ]
          }

      1	The VLAN configuration needs to specify the master name. This can be configured in the pod networks annotation.
      2	The linkInContainer parameter must be specified.

   2. Apply the YAML file by running the following command:

      $ oc apply -f vlan100-additional-network-configuration.yaml

5. Create a pod definition by using the earlier specified networks:

   1. Using the following YAML example, create a file named pod-a.yaml file:

      The manifest below includes 2 resources: Namespace with security labels Pod definition with appropriate network annotation

      apiVersion: v1
      kind: Namespace
      metadata:
        name: test-namespace
        labels:
          pod-security.kubernetes.io/enforce: privileged
          pod-security.kubernetes.io/audit: privileged
          pod-security.kubernetes.io/warn: privileged
          security.openshift.io/scc.podSecurityLabelSync: "false"
      ---
      apiVersion: v1
      kind: Pod
      metadata:
        name: nginx-pod
        namespace: test-namespace
        annotations:
          k8s.v1.cni.cncf.io/networks: '[
            {
              "name": "sriov-network",
              "namespace": "test-namespace",
              "interface": "ext0" (1)
            },
            {
              "name": "vlan-100",
              "namespace": "test-namespace",
              "interface": "ext0.100"
            }
          ]'
      spec:
        securityContext:
          runAsNonRoot: true
        containers:
          - name: nginx-container
            image: nginxinc/nginx-unprivileged:latest
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop: ["ALL"]
            ports:
              - containerPort: 80
            seccompProfile:
              type: "RuntimeDefault"

      1	The name to be used as the master for the VLAN interface.

   2. Apply the YAML file by running the following command:

      $ oc apply -f pod-a.yaml

6. Get detailed information about the nginx-pod within the test-namespace by running the following command:

   $ oc describe pods nginx-pod -n test-namespace

   Example output

   Name:         nginx-pod
   Namespace:    test-namespace
   Priority:     0
   Node:         worker-1/10.46.186.105
   Start Time:   Mon, 14 Aug 2023 16:23:13 -0400
   Labels:       <none>
   Annotations:  k8s.ovn.org/pod-networks:
                   {"default":{"ip_addresses":["10.131.0.26/23"],"mac_address":"0a:58:0a:83:00:1a","gateway_ips":["10.131.0.1"],"routes":[{"dest":"10.128.0.0...
                 k8s.v1.cni.cncf.io/network-status:
                   [{
                       "name": "ovn-kubernetes",
                       "interface": "eth0",
                       "ips": [
                           "10.131.0.26"
                       ],
                       "mac": "0a:58:0a:83:00:1a",
                       "default": true,
                       "dns": {}
                   },{
                       "name": "test-namespace/sriov-network",
                       "interface": "ext0",
                       "mac": "6e:a7:5e:3f:49:1b",
                       "dns": {},
                       "device-info": {
                           "type": "pci",
                           "version": "1.0.0",
                           "pci": {
                               "pci-address": "0000:d8:00.2"
                           }
                       }
                   },{
                       "name": "test-namespace/vlan-100",
                       "interface": "ext0.100",
                       "ips": [
                           "1.1.1.1"
                       ],
                       "mac": "6e:a7:5e:3f:49:1b",
                       "dns": {}
                   }]
                 k8s.v1.cni.cncf.io/networks:
                   [ { "name": "sriov-network", "namespace": "test-namespace", "interface": "ext0" }, { "name": "vlan-100", "namespace": "test-namespace", "i...
                 openshift.io/scc: privileged
   Status:       Running
   IP:           10.131.0.26
   IPs:
     IP:  10.131.0.26

Creating a subinterface based on a bridge master interface in a container namespace

Creating a subinterface can be applied to other types of interfaces. Follow this procedure to create a subinterface based on a bridge master interface in a container namespace.

Prerequisites

• You have installed the OpenShift CLI (oc).

• You are logged in to the OKD cluster as a user with cluster-admin privileges.

Procedure

1. Create a dedicated container namespace where you want to deploy your pod by running the following command:

   $ oc new-project test-namespace

2. Using the following YAML example, create a bridge NetworkAttachmentDefinition custom resource (CR) file named bridge-nad.yaml:

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: bridge-network
   spec:
     config: '{
       "cniVersion": "0.4.0",
       "name": "bridge-network",
       "type": "bridge",
       "bridge": "br-001",
       "isGateway": true,
       "ipMasq": true,
       "hairpinMode": true,
       "ipam": {
         "type": "host-local",
         "subnet": "10.0.0.0/24",
         "routes": [{"dst": "0.0.0.0/0"}]
       }
     }'

3. Run the following command to apply the NetworkAttachmentDefinition CR to your OKD cluster:

   $ oc apply -f bridge-nad.yaml

4. Verify that the NetworkAttachmentDefinition CR has been created successfully by running the following command:

   $ oc get network-attachment-definitions

   Example output

   NAME             AGE
   bridge-network   15s

5. Using the following YAML example, create a file named ipvlan-additional-network-configuration.yaml for the IPVLAN additional network configuration:

   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     name: ipvlan-net
     namespace: test-namespace
   spec:
     config: '{
       "cniVersion": "0.3.1",
       "name": "ipvlan-net",
       "type": "ipvlan",
       "master": "ext0", (1)
       "mode": "l3",
       "linkInContainer": true, (2)
       "ipam": {"type": "whereabouts", "ipRanges": [{"range": "10.0.0.0/24"}]}
     }'

   1	Specifies the ethernet interface to associate with the network attachment. This is subsequently configured in the pod networks annotation.
   2	Specifies that the master interface is in the container network namespace.

6. Apply the YAML file by running the following command:

   $ oc apply -f ipvlan-additional-network-configuration.yaml

7. Verify that the NetworkAttachmentDefinition CR has been created successfully by running the following command:

   $ oc get network-attachment-definitions

   Example output

   NAME             AGE
   bridge-network   87s
   ipvlan-net       9s

8. Using the following YAML example, create a file named pod-a.yaml for the pod definition:

   apiVersion: v1
   kind: Pod
   metadata:
     name: pod-a
     namespace: test-namespace
     annotations:
       k8s.v1.cni.cncf.io/networks: '[
         {
           "name": "bridge-network",
           "interface": "ext0" (1)
         },
         {
           "name": "ipvlan-net",
           "interface": "ext1"
         }
       ]'
   spec:
     securityContext:
       runAsNonRoot: true
       seccompProfile:
         type: RuntimeDefault
     containers:
     - name: test-pod
       image: quay.io/openshifttest/hello-sdn@sha256:c89445416459e7adea9a5a416b3365ed3d74f2491beb904d61dc8d1eb89a72a4
       securityContext:
         allowPrivilegeEscalation: false
         capabilities:
           drop: [ALL]

   1	Specifies the name to be used as the master for the IPVLAN interface.

9. Apply the YAML file by running the following command:

   $ oc apply -f pod-a.yaml

10. Verify that the pod is running by using the following command:

    $ oc get pod -n test-namespace

    Example output

    NAME    READY   STATUS    RESTARTS   AGE
    pod-a   1/1     Running   0          2m36s

11. Show network interface information about the pod-a resource within the test-namespace by running the following command:

    $ oc exec -n test-namespace pod-a -- ip a

    Example output

    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
       link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
       inet 127.0.0.1/8 scope host lo
          valid_lft forever preferred_lft forever
       inet6 ::1/128 scope host
          valid_lft forever preferred_lft forever
    3: eth0@if105: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
       link/ether 0a:58:0a:d9:00:5d brd ff:ff:ff:ff:ff:ff link-netnsid 0
       inet 10.217.0.93/23 brd 10.217.1.255 scope global eth0
          valid_lft forever preferred_lft forever
       inet6 fe80::488b:91ff:fe84:a94b/64 scope link
          valid_lft forever preferred_lft forever
    4: ext0@if107: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
       link/ether be:da:bd:7e:f4:37 brd ff:ff:ff:ff:ff:ff link-netnsid 0
       inet 10.0.0.2/24 brd 10.0.0.255 scope global ext0
          valid_lft forever preferred_lft forever
       inet6 fe80::bcda:bdff:fe7e:f437/64 scope link
          valid_lft forever preferred_lft forever
    5: ext1@ext0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
       link/ether be:da:bd:7e:f4:37 brd ff:ff:ff:ff:ff:ff
       inet 10.0.0.1/24 brd 10.0.0.255 scope global ext1
          valid_lft forever preferred_lft forever
       inet6 fe80::beda:bd00:17e:f437/64 scope link
          valid_lft forever preferred_lft forever

    This output shows that the network interface ext1 is associated with the physical interface ext0.