Configuring interface-level network sysctl settings and all-multicast mode for SR-IOV networks

As a cluster administrator, you can change interface-level network sysctls and several interface attributes such as promiscuous mode, all-multicast mode, MTU, and MAC address by using the tuning Container Network Interface (CNI) meta plugin for a pod connected to a SR-IOV network device.

Labeling nodes with an SR-IOV enabled NIC

If you want to enable SR-IOV on only SR-IOV capable nodes there are a couple of ways to do this:

  1. Install the Node Feature Discovery (NFD) Operator. NFD detects the presence of SR-IOV enabled NICs and labels the nodes with node.alpha.kubernetes-incubator.io/nfd-network-sriov.capable = true.

  2. Examine the SriovNetworkNodeState CR for each node. The interfaces stanza includes a list of all of the SR-IOV devices discovered by the SR-IOV Network Operator on the worker node. Label each node with feature.node.kubernetes.io/network-sriov.capable: "true" by using the following command:

    1. $ oc label node <node_name> feature.node.kubernetes.io/network-sriov.capable="true"

    You can label the nodes with whatever name you want.

Setting one sysctl flag

You can set interface-level network sysctl settings for a pod connected to a SR-IOV network device.

In this example, net.ipv4.conf.IFNAME.accept_redirects is set to 1 on the created virtual interfaces.

The sysctl-tuning-test is a namespace used in this example.

  • Use the following command to create the sysctl-tuning-test namespace:

    1. $ oc create namespace sysctl-tuning-test

Setting one sysctl flag on nodes with SR-IOV network devices

The SR-IOV Network Operator adds the SriovNetworkNodePolicy.sriovnetwork.openshift.io custom resource definition (CRD) to OKD. You can configure an SR-IOV network device by creating a SriovNetworkNodePolicy custom resource (CR).

When applying the configuration specified in a SriovNetworkNodePolicy object, the SR-IOV Operator might drain and reboot the nodes.

It can take several minutes for a configuration change to apply.

Follow this procedure to create a SriovNetworkNodePolicy custom resource (CR).

Procedure

  1. Create an SriovNetworkNodePolicy custom resource (CR). For example, save the following YAML as the file policyoneflag-sriov-node-network.yaml:

    1. apiVersion: sriovnetwork.openshift.io/v1
    2. kind: SriovNetworkNodePolicy
    3. metadata:
    4.   name: policyoneflag (1)
    5.   namespace: openshift-sriov-network-operator (2)
    6. spec:
    7.   resourceName: policyoneflag (3)
    8.   nodeSelector: (4)
    9.     feature.node.kubernetes.io/network-sriov.capable="true"
    10.   priority: 10 (5)
    11.   numVfs: 5 (6)
    12.   nicSelector: (7)
    13.     pfNames: ["ens5"] (8)
    14.   deviceType: "netdevice" (9)
    15.   isRdma: false (10)
    1The name for the custom resource object.
    2The namespace where the SR-IOV Network Operator is installed.
    3The resource name of the SR-IOV network device plugin. You can create multiple SR-IOV network node policies for a resource name.
    4The node selector specifies the nodes to configure. Only SR-IOV network devices on the selected nodes are configured. The SR-IOV Container Network Interface (CNI) plugin and device plugin are deployed on selected nodes only.
    5Optional: The priority is an integer value between 0 and 99. A smaller value receives higher priority. For example, a priority of 10 is a higher priority than 99. The default value is 99.
    6The number of the virtual functions (VFs) to create for the SR-IOV physical network device. For an Intel network interface controller (NIC), the number of VFs cannot be larger than the total VFs supported by the device. For a Mellanox NIC, the number of VFs cannot be larger than 128.
    7The NIC selector identifies the device for the Operator to configure. You do not have to specify values for all the parameters. It is recommended to identify the network device with enough precision to avoid selecting a device unintentionally. If you specify rootDevices, you must also specify a value for vendor, deviceID, or pfNames. If you specify both pfNames and rootDevices at the same time, ensure that they refer to the same device. If you specify a value for netFilter, then you do not need to specify any other parameter because a network ID is unique.
    8Optional: An array of one or more physical function (PF) names for the device.
    9Optional: The driver type for the virtual functions. The only allowed value is netdevice. For a Mellanox NIC to work in DPDK mode on bare metal nodes, set isRdma to true.
    10Optional: Configures whether to enable remote direct memory access (RDMA) mode. The default value is false. If the isRdma parameter is set to true, you can continue to use the RDMA-enabled VF as a normal network device. A device can be used in either mode. Set isRdma to true and additionally set needVhostNet to true to configure a Mellanox NIC for use with Fast Datapath DPDK applications.

    The vfio-pci driver type is not supported.

  2. Create the SriovNetworkNodePolicy object:

    1. $ oc create -f policyoneflag-sriov-node-network.yaml

    After applying the configuration update, all the pods in sriov-network-operator namespace change to the Running status.

  3. To verify that the SR-IOV network device is configured, enter the following command. Replace <node_name> with the name of a node with the SR-IOV network device that you just configured.

    1. $ oc get sriovnetworknodestates -n openshift-sriov-network-operator <node_name> -o jsonpath='{.status.syncStatus}'

    Example output

    1. Succeeded

Configuring sysctl on a SR-IOV network

You can set interface specific sysctl settings on virtual interfaces created by SR-IOV by adding the tuning configuration to the optional metaPlugins parameter of the SriovNetwork resource.

The SR-IOV Network Operator manages additional network definitions. When you specify an additional SR-IOV network to create, the SR-IOV Network Operator creates the NetworkAttachmentDefinition custom resource (CR) automatically.

Do not edit NetworkAttachmentDefinition custom resources that the SR-IOV Network Operator manages. Doing so might disrupt network traffic on your additional network.

To change the interface-level network net.ipv4.conf.IFNAME.accept_redirects sysctl settings, create an additional SR-IOV network with the Container Network Interface (CNI) tuning plugin.

Prerequisites

  • Install the OKD CLI (oc).

  • Log in to the OKD cluster as a user with cluster-admin privileges.

Procedure

  1. Create the SriovNetwork custom resource (CR) for the additional SR-IOV network attachment and insert the metaPlugins configuration, as in the following example CR. Save the YAML as the file sriov-network-interface-sysctl.yaml.

    1. apiVersion: sriovnetwork.openshift.io/v1
    2. kind: SriovNetwork
    3. metadata:
    4.   name: onevalidflag (1)
    5.   namespace: openshift-sriov-network-operator (2)
    6. spec:
    7.   resourceName: policyoneflag (3)
    8.   networkNamespace: sysctl-tuning-test (4)
    9.   ipam: '{ "type": "static" }' (5)
    10.   capabilities: '{ "mac": true, "ips": true }' (6)
    11.   metaPlugins : | (7)
    12.     {
    13.       "type": "tuning",
    14.       "capabilities":{
    15.         "mac":true
    16.       },
    17.       "sysctl":{
    18.          "net.ipv4.conf.IFNAME.accept_redirects": "1"
    19.       }
    20.     }
    1A name for the object. The SR-IOV Network Operator creates a NetworkAttachmentDefinition object with same name.
    2The namespace where the SR-IOV Network Operator is installed.
    3The value for the spec.resourceName parameter from the SriovNetworkNodePolicy object that defines the SR-IOV hardware for this additional network.
    4The target namespace for the SriovNetwork object. Only pods in the target namespace can attach to the additional network.
    5A configuration object for the IPAM CNI plugin as a YAML block scalar. The plugin manages IP address assignment for the attachment definition.
    6Optional: Set capabilities for the additional network. You can specify “{ “ips”: true }” to enable IP address support or “{ “mac”: true }” to enable MAC address support.
    7Optional: The metaPlugins parameter is used to add additional capabilities to the device. In this use case set the type field to tuning. Specify the interface-level network sysctl you want to set in the sysctl field.

  2. Create the SriovNetwork resource:

    1. $ oc create -f sriov-network-interface-sysctl.yaml

Verifying that the NetworkAttachmentDefinition CR is successfully created

  • Confirm that the SR-IOV Network Operator created the NetworkAttachmentDefinition CR by running the following command:

    1. $ oc get network-attachment-definitions -n <namespace> (1)
    1Replace <namespace> with the value for networkNamespace that you specified in the SriovNetwork object. For example, sysctl-tuning-test.

    Example output

    1. NAME                                  AGE
    2. onevalidflag                          14m

    There might be a delay before the SR-IOV Network Operator creates the CR.

Verifying that the additional SR-IOV network attachment is successful

To verify that the tuning CNI is correctly configured and the additional SR-IOV network attachment is attached, do the following:

  1. Create a Pod CR. Save the following YAML as the file examplepod.yaml:

    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4.   name: tunepod
    5.   namespace: sysctl-tuning-test
    6.   annotations:
    7.     k8s.v1.cni.cncf.io/networks: |-
    8.       [
    9.         {
    10.           "name": "onevalidflag",  (1)
    11.           "mac": "0a:56:0a:83:04:0c", (2)
    12.           "ips": ["10.100.100.200/24"] (3)
    13.        }
    14.       ]
    15. spec:
    16.   containers:
    17.   - name: podexample
    18.     image: centos
    19.     command: ["/bin/bash", "-c", "sleep INF"]
    20.     securityContext:
    21.       runAsUser: 2000
    22.       runAsGroup: 3000
    23.       allowPrivilegeEscalation: false
    24.       capabilities:
    25.         drop: ["ALL"]
    26.   securityContext:
    27.     runAsNonRoot: true
    28.     seccompProfile:
    29.       type: RuntimeDefault
    1The name of the SR-IOV network attachment definition CR.
    2Optional: The MAC address for the SR-IOV device that is allocated from the resource type defined in the SR-IOV network attachment definition CR. To use this feature, you also must specify { “mac”: true } in the SriovNetwork object.
    3Optional: IP addresses for the SR-IOV device that are allocated from the resource type defined in the SR-IOV network attachment definition CR. Both IPv4 and IPv6 addresses are supported. To use this feature, you also must specify { “ips”: true } in the SriovNetwork object.

  2. Create the Pod CR:

    1. $ oc apply -f examplepod.yaml

  3. Verify that the pod is created by running the following command:

    1. $ oc get pod -n sysctl-tuning-test

    Example output

    1. NAME      READY   STATUS    RESTARTS   AGE
    2. tunepod   1/1     Running   0          47s

  4. Log in to the pod by running the following command:

    1. $ oc rsh -n sysctl-tuning-test tunepod

  5. Verify the values of the configured sysctl flag. Find the value net.ipv4.conf.IFNAME.accept_redirects by running the following command::

    1. $ sysctl net.ipv4.conf.net1.accept_redirects

    Example output

    1. net.ipv4.conf.net1.accept_redirects = 1

Configuring sysctl settings for pods associated with bonded SR-IOV interface flag

You can set interface-level network sysctl settings for a pod connected to a bonded SR-IOV network device.

In this example, the specific network interface-level sysctl settings that can be configured are set on the bonded interface.

The sysctl-tuning-test is a namespace used in this example.

  • Use the following command to create the sysctl-tuning-test namespace:

    1. $ oc create namespace sysctl-tuning-test

Setting all sysctl flag on nodes with bonded SR-IOV network devices

The SR-IOV Network Operator adds the SriovNetworkNodePolicy.sriovnetwork.openshift.io custom resource definition (CRD) to OKD. You can configure an SR-IOV network device by creating a SriovNetworkNodePolicy custom resource (CR).

When applying the configuration specified in a SriovNetworkNodePolicy object, the SR-IOV Operator might drain the nodes, and in some cases, reboot nodes.

It might take several minutes for a configuration change to apply.

Follow this procedure to create a SriovNetworkNodePolicy custom resource (CR).

Procedure

  1. Create an SriovNetworkNodePolicy custom resource (CR). Save the following YAML as the file policyallflags-sriov-node-network.yaml. Replace policyallflags with the name for the configuration.

    1. apiVersion: sriovnetwork.openshift.io/v1
    2. kind: SriovNetworkNodePolicy
    3. metadata:
    4.   name: policyallflags (1)
    5.   namespace: openshift-sriov-network-operator (2)
    6. spec:
    7.   resourceName: policyallflags (3)
    8.   nodeSelector: (4)
    9.     node.alpha.kubernetes-incubator.io/nfd-network-sriov.capable = `true`
    10.   priority: 10 (5)
    11.   numVfs: 5 (6)
    12.   nicSelector: (7)
    13.     pfNames: ["ens1f0"]  (8)
    14.   deviceType: "netdevice" (9)
    15.   isRdma: false (10)
    1The name for the custom resource object.
    2The namespace where the SR-IOV Network Operator is installed.
    3The resource name of the SR-IOV network device plugin. You can create multiple SR-IOV network node policies for a resource name.
    4The node selector specifies the nodes to configure. Only SR-IOV network devices on the selected nodes are configured. The SR-IOV Container Network Interface (CNI) plugin and device plugin are deployed on selected nodes only.
    5Optional: The priority is an integer value between 0 and 99. A smaller value receives higher priority. For example, a priority of 10 is a higher priority than 99. The default value is 99.
    6The number of virtual functions (VFs) to create for the SR-IOV physical network device. For an Intel network interface controller (NIC), the number of VFs cannot be larger than the total VFs supported by the device. For a Mellanox NIC, the number of VFs cannot be larger than 128.
    7The NIC selector identifies the device for the Operator to configure. You do not have to specify values for all the parameters. It is recommended to identify the network device with enough precision to avoid selecting a device unintentionally. If you specify rootDevices, you must also specify a value for vendor, deviceID, or pfNames. If you specify both pfNames and rootDevices at the same time, ensure that they refer to the same device. If you specify a value for netFilter, then you do not need to specify any other parameter because a network ID is unique.
    8Optional: An array of one or more physical function (PF) names for the device.
    9Optional: The driver type for the virtual functions. The only allowed value is netdevice. For a Mellanox NIC to work in DPDK mode on bare metal nodes, set isRdma to true.
    10Optional: Configures whether to enable remote direct memory access (RDMA) mode. The default value is false. If the isRdma parameter is set to true, you can continue to use the RDMA-enabled VF as a normal network device. A device can be used in either mode. Set isRdma to true and additionally set needVhostNet to true to configure a Mellanox NIC for use with Fast Datapath DPDK applications.

    The vfio-pci driver type is not supported.

  2. Create the SriovNetworkNodePolicy object:

    1. $ oc create -f policyallflags-sriov-node-network.yaml

    After applying the configuration update, all the pods in sriov-network-operator namespace change to the Running status.

  3. To verify that the SR-IOV network device is configured, enter the following command. Replace <node_name> with the name of a node with the SR-IOV network device that you just configured.

    1. $ oc get sriovnetworknodestates -n openshift-sriov-network-operator <node_name> -o jsonpath='{.status.syncStatus}'

    Example output

    1. Succeeded

Configuring sysctl on a bonded SR-IOV network

You can set interface specific sysctl settings on a bonded interface created from two SR-IOV interfaces. Do this by adding the tuning configuration to the optional Plugins parameter of the bond network attachment definition.

Do not edit NetworkAttachmentDefinition custom resources that the SR-IOV Network Operator manages. Doing so might disrupt network traffic on your additional network.

To change specific interface-level network sysctl settings create the SriovNetwork custom resource (CR) with the Container Network Interface (CNI) tuning plugin by using the following procedure.

Prerequisites

  • Install the OKD CLI (oc).

  • Log in to the OKD cluster as a user with cluster-admin privileges.

Procedure

  1. Create the SriovNetwork custom resource (CR) for the bonded interface as in the following example CR. Save the YAML as the file sriov-network-attachment.yaml.

    1. apiVersion: sriovnetwork.openshift.io/v1
    2. kind: SriovNetwork
    3. metadata:
    4.   name: allvalidflags (1)
    5.   namespace: openshift-sriov-network-operator (2)
    6. spec:
    7.   resourceName: policyallflags (3)
    8.   networkNamespace: sysctl-tuning-test (4)
    9.   capabilities: '{ "mac": true, "ips": true }' (5)
    1A name for the object. The SR-IOV Network Operator creates a NetworkAttachmentDefinition object with same name.
    2The namespace where the SR-IOV Network Operator is installed.
    3The value for the spec.resourceName parameter from the SriovNetworkNodePolicy object that defines the SR-IOV hardware for this additional network.
    4The target namespace for the SriovNetwork object. Only pods in the target namespace can attach to the additional network.
    5Optional: The capabilities to configure for this additional network. You can specify “{ “ips”: true }” to enable IP address support or “{ “mac”: true }” to enable MAC address support.

  2. Create the SriovNetwork resource:

    1. $ oc create -f sriov-network-attachment.yaml

  3. Create a bond network attachment definition as in the following example CR. Save the YAML as the file sriov-bond-network-interface.yaml.

    1. apiVersion: "k8s.cni.cncf.io/v1"
    2. kind: NetworkAttachmentDefinition
    3. metadata:
    4.   name: bond-sysctl-network
    5.   namespace: sysctl-tuning-test
    6. spec:
    7.   config: '{
    8.   "cniVersion":"0.4.0",
    9.   "name":"bound-net",
    10.   "plugins":[
    11.     {
    12.       "type":"bond", (1)
    13.       "mode": "active-backup", (2)
    14.       "failOverMac": 1, (3)
    15.       "linksInContainer": true, (4)
    16.       "miimon": "100",
    17.       "links": [ (5)
    18.         {"name": "net1"},
    19.         {"name": "net2"}
    20.       ],
    21.       "ipam":{ (6)
    22.         "type":"static"
    23.       }
    24.     },
    25.     {
    26.       "type":"tuning", (7)
    27.       "capabilities":{
    28.         "mac":true
    29.       },
    30.       "sysctl":{
    31.         "net.ipv4.conf.IFNAME.accept_redirects": "0",
    32.         "net.ipv4.conf.IFNAME.accept_source_route": "0",
    33.         "net.ipv4.conf.IFNAME.disable_policy": "1",
    34.         "net.ipv4.conf.IFNAME.secure_redirects": "0",
    35.         "net.ipv4.conf.IFNAME.send_redirects": "0",
    36.         "net.ipv6.conf.IFNAME.accept_redirects": "0",
    37.         "net.ipv6.conf.IFNAME.accept_source_route": "1",
    38.         "net.ipv6.neigh.IFNAME.base_reachable_time_ms": "20000",
    39.         "net.ipv6.neigh.IFNAME.retrans_time_ms": "2000"
    40.       }
    41.     }
    42.   ]
    43. }'
    1The type is bond.
    2The mode attribute specifies the bonding mode. The bonding modes supported are:

    • balance-rr - 0

    • active-backup - 1

    • balance-xor - 2

      For balance-rr or balance-xor modes, you must set the trust mode to on for the SR-IOV virtual function.

    3The failover attribute is mandatory for active-backup mode.
    4The linksInContainer=true flag informs the Bond CNI that the required interfaces are to be found inside the container. By default, Bond CNI looks for these interfaces on the host which does not work for integration with SRIOV and Multus.
    5The links section defines which interfaces will be used to create the bond. By default, Multus names the attached interfaces as: “net”, plus a consecutive number, starting with one.
    6A configuration object for the IPAM CNI plugin as a YAML block scalar. The plugin manages IP address assignment for the attachment definition. In this pod example IP addresses are configured manually, so in this case,ipam is set to static.
    7Add additional capabilities to the device. For example, set the type field to tuning. Specify the interface-level network sysctl you want to set in the sysctl field. This example sets all interface-level network sysctl settings that can be set.

  4. Create the bond network attachment resource:

    1. $ oc create -f sriov-bond-network-interface.yaml

Verifying that the NetworkAttachmentDefinition CR is successfully created

  • Confirm that the SR-IOV Network Operator created the NetworkAttachmentDefinition CR by running the following command:

    1. $ oc get network-attachment-definitions -n <namespace> (1)
    1Replace <namespace> with the networkNamespace that you specified when configuring the network attachment, for example, sysctl-tuning-test.

    Example output

    1. NAME                          AGE
    2. bond-sysctl-network           22m
    3. allvalidflags                 47m

    There might be a delay before the SR-IOV Network Operator creates the CR.

Verifying that the additional SR-IOV network resource is successful

To verify that the tuning CNI is correctly configured and the additional SR-IOV network attachment is attached, do the following:

  1. Create a Pod CR. For example, save the following YAML as the file examplepod.yaml:

    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4.   name: tunepod
    5.   namespace: sysctl-tuning-test
    6.   annotations:
    7.     k8s.v1.cni.cncf.io/networks: |-
    8.       [
    9.         {"name": "allvalidflags"}, (1)
    10.         {"name": "allvalidflags"},
    11.         {
    12.           "name": "bond-sysctl-network",
    13.           "interface": "bond0",
    14.           "mac": "0a:56:0a:83:04:0c", (2)
    15.           "ips": ["10.100.100.200/24"] (3)
    16.        }
    17.       ]
    18. spec:
    19.   containers:
    20.   - name: podexample
    21.     image: centos
    22.     command: ["/bin/bash", "-c", "sleep INF"]
    23.     securityContext:
    24.       runAsUser: 2000
    25.       runAsGroup: 3000
    26.       allowPrivilegeEscalation: false
    27.       capabilities:
    28.         drop: ["ALL"]
    29.   securityContext:
    30.     runAsNonRoot: true
    31.     seccompProfile:
    32.       type: RuntimeDefault
    1The name of the SR-IOV network attachment definition CR.
    2Optional: The MAC address for the SR-IOV device that is allocated from the resource type defined in the SR-IOV network attachment definition CR. To use this feature, you also must specify { “mac”: true } in the SriovNetwork object.
    3Optional: IP addresses for the SR-IOV device that are allocated from the resource type defined in the SR-IOV network attachment definition CR. Both IPv4 and IPv6 addresses are supported. To use this feature, you also must specify { “ips”: true } in the SriovNetwork object.

  2. Apply the YAML:

    1. $ oc apply -f examplepod.yaml

  3. Verify that the pod is created by running the following command:

    1. $ oc get pod -n sysctl-tuning-test

    Example output

    1. NAME      READY   STATUS    RESTARTS   AGE
    2. tunepod   1/1     Running   0          47s

  4. Log in to the pod by running the following command:

    1. $ oc rsh -n sysctl-tuning-test tunepod

  5. Verify the values of the configured sysctl flag. Find the value net.ipv6.neigh.IFNAME.base_reachable_time_ms by running the following command::

    1. $ sysctl net.ipv6.neigh.bond0.base_reachable_time_ms

    Example output

    1. net.ipv6.neigh.bond0.base_reachable_time_ms = 20000

About all-multicast mode

Enabling all-multicast mode, particularly in the context of rootless applications, is critical. If you do not enable this mode, you would be required to grant the NET_ADMIN capability to the pod’s Security Context Constraints (SCC). If you were to allow the NET_ADMIN capability to grant the pod privileges to make changes that extend beyond its specific requirements, you could potentially expose security vulnerabilities.

The tuning CNI plugin supports changing several interface attributes, including all-multicast mode. By enabling this mode, you can allow applications running on Virtual Functions (VFs) that are configured on a SR-IOV network device to receive multicast traffic from applications on other VFs, whether attached to the same or different physical functions.

Enabling the all-multicast mode on an SR-IOV network

You can enable the all-multicast mode on an SR-IOV interface by:

  • Adding the tuning configuration to the metaPlugins parameter of the SriovNetwork resource

  • Setting the allmulti field to true in the tuning configuration

    Ensure that you create the virtual function (VF) with trust enabled.

The SR-IOV Network Operator manages additional network definitions. When you specify an additional SR-IOV network to create, the SR-IOV Network Operator creates the NetworkAttachmentDefinition custom resource (CR) automatically.

Do not edit NetworkAttachmentDefinition custom resources that the SR-IOV Network Operator manages. Doing so might disrupt network traffic on your additional network.

Enable the all-multicast mode on a SR-IOV network by following this guidance.

Prerequisites

  • You have installed the OKD CLI (oc).

  • You are logged in to the OKD cluster as a user with cluster-admin privileges.

  • You have installed the SR-IOV Network Operator.

  • You have configured an appropriate SriovNetworkNodePolicy object.

Procedure

  1. Create a YAML file with the following settings that defines a SriovNetworkNodePolicy object for a Mellanox ConnectX-5 device. Save the YAML file as sriovnetpolicy-mlx.yaml.

    1. apiVersion: sriovnetwork.openshift.io/v1
    2. kind: SriovNetworkNodePolicy
    3. metadata:
    4.   name: sriovnetpolicy-mlx
    5.   namespace: openshift-sriov-network-operator
    6. spec:
    7.   deviceType: netdevice
    8.   nicSelector:
    9.     deviceID: "1017"
    10.     pfNames:
    11.       - ens8f0np0#0-9
    12.     rootDevices:
    13.       - 0000:d8:00.0
    14.     vendor: "15b3"
    15.   nodeSelector:
    16.     feature.node.kubernetes.io/network-sriov.capable: "true"
    17.   numVfs: 10
    18.   priority: 99
    19.   resourceName: resourcemlx

  2. Optional: If the SR-IOV capable cluster nodes are not already labeled, add the SriovNetworkNodePolicy.Spec.NodeSelector label. For more information about labeling nodes, see “Understanding how to update labels on nodes”.

  3. Create the SriovNetworkNodePolicy object by running the following command:

    1. $ oc create -f sriovnetpolicy-mlx.yaml

    After applying the configuration update, all the pods in the sriov-network-operator namespace automatically move to a Running status.

  4. Create the enable-allmulti-test namespace by running the following command:

    1. $ oc create namespace enable-allmulti-test

  5. Create the SriovNetwork custom resource (CR) for the additional SR-IOV network attachment and insert the metaPlugins configuration, as in the following example CR YAML, and save the file as sriov-enable-all-multicast.yaml.

    1. apiVersion: sriovnetwork.openshift.io/v1
    2. kind: SriovNetwork
    3. metadata:
    4.   name: enableallmulti (1)
    5.   namespace: openshift-sriov-network-operator (2)
    6. spec:
    7.   resourceName: enableallmulti (3)
    8.   networkNamespace: enable-allmulti-test (4)
    9.   ipam: '{ "type": "static" }' (5)
    10.   capabilities: '{ "mac": true, "ips": true }' (6)
    11.   trust: "on" (7)
    12.   metaPlugins : | (8)
    13.     {
    14.       "type": "tuning",
    15.       "capabilities":{
    16.         "mac":true
    17.       },
    18.       "allmulti": true
    19.       }
    20.     }
    1Specify a name for the object. The SR-IOV Network Operator creates a NetworkAttachmentDefinition object with the same name.
    2Specify the namespace where the SR-IOV Network Operator is installed.
    3Specify a value for the spec.resourceName parameter from the SriovNetworkNodePolicy object that defines the SR-IOV hardware for this additional network.
    4Specify the target namespace for the SriovNetwork object. Only pods in the target namespace can attach to the additional network.
    5Specify a configuration object for the IPAM CNI plugin as a YAML block scalar. The plugin manages IP address assignment for the attachment definition.
    6Optional: Set capabilities for the additional network. You can specify “{ “ips”: true }” to enable IP address support or “{ “mac”: true }” to enable MAC address support.
    7Specify the trust mode of the virtual function. This must be set to “on”.
    8Add more capabilities to the device by using the metaPlugins parameter. In this use case, set the type field to tuning, and add the allmulti field and set it to true.

  6. Create the SriovNetwork resource by running the following command:

    1. $ oc create -f sriov-enable-all-multicast.yaml

Verification of the NetworkAttachmentDefinition CR

  • Confirm that the SR-IOV Network Operator created the NetworkAttachmentDefinition CR by running the following command:

    1. $ oc get network-attachment-definitions -n <namespace> (1)
    1Replace <namespace> with the value for networkNamespace that you specified in the SriovNetwork object. For this example, that is enable-allmulti-test.

    Example output

    1. NAME                                  AGE
    2. enableallmulti                        14m

    There might be a delay before the SR-IOV Network Operator creates the CR.

    1. Display information about the SR-IOV network resources by running the following command:

      1. $ oc get sriovnetwork -n openshift-sriov-network-operator

Verification of the additional SR-IOV network attachment

To verify that the tuning CNI is correctly configured and that the additional SR-IOV network attachment is attached, follow these steps:

  1. Create a Pod CR. Save the following sample YAML in a file named examplepod.yaml:

    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4.   name: samplepod
    5.   namespace: enable-allmulti-test
    6.   annotations:
    7.     k8s.v1.cni.cncf.io/networks: |-
    8.       [
    9.         {
    10.           "name": "enableallmulti",  (1)
    11.           "mac": "0a:56:0a:83:04:0c", (2)
    12.           "ips": ["10.100.100.200/24"] (3)
    13.        }
    14.       ]
    15. spec:
    16.   containers:
    17.   - name: podexample
    18.     image: centos
    19.     command: ["/bin/bash", "-c", "sleep INF"]
    20.     securityContext:
    21.       runAsUser: 2000
    22.       runAsGroup: 3000
    23.       allowPrivilegeEscalation: false
    24.       capabilities:
    25.         drop: ["ALL"]
    26.   securityContext:
    27.     runAsNonRoot: true
    28.     seccompProfile:
    29.       type: RuntimeDefault
    1Specify the name of the SR-IOV network attachment definition CR.
    2Optional: Specify the MAC address for the SR-IOV device that is allocated from the resource type defined in the SR-IOV network attachment definition CR. To use this feature, you also must specify {“mac”: true} in the SriovNetwork object.
    3Optional: Specify the IP addresses for the SR-IOV device that are allocated from the resource type defined in the SR-IOV network attachment definition CR. Both IPv4 and IPv6 addresses are supported. To use this feature, you also must specify { “ips”: true } in the SriovNetwork object.

  2. Create the Pod CR by running the following command:

    1. $ oc apply -f examplepod.yaml

  3. Verify that the pod is created by running the following command:

    1. $ oc get pod -n enable-allmulti-test

    Example output

    1. NAME       READY   STATUS    RESTARTS   AGE
    2. samplepod  1/1     Running   0          47s

  4. Log in to the pod by running the following command:

    1. $ oc rsh -n enable-allmulti-test samplepod

  5. List all the interfaces associated with the pod by running the following command:

    1. sh-4.4# ip link

    Example output

    1. 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    2.     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    3. 2: eth0@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8901 qdisc noqueue state UP mode DEFAULT group default
    4.     link/ether 0a:58:0a:83:00:10 brd ff:ff:ff:ff:ff:ff link-netnsid 0 (1)
    5. 3: net1@if24: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    6.     link/ether ee:9b:66:a4:ec:1d brd ff:ff:ff:ff:ff:ff link-netnsid 0 (2)
    1eth0@if22 is the primary interface
    2net1@if24 is the secondary interface configured with the network-attachment-definition that supports the all-multicast mode (ALLMULTI flag)