Connecting a virtual machine to an SR-IOV network

You can connect a virtual machine (VM) to a Single Root I/O Virtualization (SR-IOV) network by performing the following steps:

1. Configure an SR-IOV network device.

2. Configure an SR-IOV network.

3. Connect the VM to the SR-IOV network.

Prerequisites

• You must have enabled global SR-IOV and VT-d settings in the firmware for the host.

• You must have installed the SR-IOV Network Operator.

Configuring SR-IOV network devices

The SR-IOV Network Operator adds the SriovNetworkNodePolicy.sriovnetwork.openshift.io CustomResourceDefinition to OKD. You can configure an SR-IOV network device by creating a SriovNetworkNodePolicy custom resource (CR).

Prerequisites

• You installed the OpenShift CLI (oc).

• You have access to the cluster as a user with the cluster-admin role.

• You have installed the SR-IOV Network Operator.

• You have enough available nodes in your cluster to handle the evicted workload from drained nodes.

• You have not selected any control plane nodes for SR-IOV network device configuration.

Procedure

1. Create an SriovNetworkNodePolicy object, and then save the YAML in the <name>-sriov-node-network.yaml file. Replace <name> with the name for this configuration.

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: <name> (1)
     namespace: openshift-sriov-network-operator (2)
   spec:
     resourceName: <sriov_resource_name> (3)
     nodeSelector:
       feature.node.kubernetes.io/network-sriov.capable: "true" (4)
     priority: <priority> (5)
     mtu: <mtu> (6)
     numVfs: <num> (7)
     nicSelector: (8)
       vendor: "<vendor_code>" (9)
       deviceID: "<device_id>" (10)
       pfNames: ["<pf_name>", ...] (11)
       rootDevices: ["<pci_bus_id>", "..."] (12)
     deviceType: vfio-pci (13)
     isRdma: false (14)

   1	Specify a name for the CR object.
   2	Specify the namespace where the SR-IOV Operator is installed.
   3	Specify the resource name of the SR-IOV device plugin. You can create multiple SriovNetworkNodePolicy objects for a resource name.
   4	Specify the node selector to select which nodes are configured. Only SR-IOV network devices on selected nodes are configured. The SR-IOV Container Network Interface (CNI) plugin and device plugin are deployed only on selected nodes.
   5	Optional: Specify an integer value between 0 and 99. A smaller number gets higher priority, so a priority of 10 is higher than a priority of 99. The default value is 99.
   6	Optional: Specify a value for the maximum transmission unit (MTU) of the virtual function. The maximum MTU value can vary for different NIC models.
   7	Specify the number of the virtual functions (VF) to create for the SR-IOV physical network device. For an Intel network interface controller (NIC), the number of VFs cannot be larger than the total VFs supported by the device. For a Mellanox NIC, the number of VFs cannot be larger than 128.
   8	The nicSelector mapping selects the Ethernet device for the Operator to configure. You do not need to specify values for all the parameters. It is recommended to identify the Ethernet adapter with enough precision to minimize the possibility of selecting an Ethernet device unintentionally. If you specify rootDevices, you must also specify a value for vendor, deviceID, or pfNames. If you specify both pfNames and rootDevices at the same time, ensure that they point to an identical device.
   9	Optional: Specify the vendor hex code of the SR-IOV network device. The only allowed values are either 8086 or 15b3.
   10	Optional: Specify the device hex code of SR-IOV network device. The only allowed values are 158b, 1015, 1017.
   11	Optional: The parameter accepts an array of one or more physical function (PF) names for the Ethernet device.
   12	The parameter accepts an array of one or more PCI bus addresses for the physical function of the Ethernet device. Provide the address in the following format: 0000:02:00.1.
   13	The vfio-pci driver type is required for virtual functions in OKD Virtualization.
   14	Optional: Specify whether to enable remote direct memory access (RDMA) mode. For a Mellanox card, set isRdma to false. The default value is false.

   If isRDMA flag is set to true, you can continue to use the RDMA enabled VF as a normal network device. A device can be used in either mode.

2. Optional: Label the SR-IOV capable cluster nodes with SriovNetworkNodePolicy.Spec.NodeSelector if they are not already labeled. For more information about labeling nodes, see “Understanding how to update labels on nodes”.

3. Create the SriovNetworkNodePolicy object:

   $ oc create -f <name>-sriov-node-network.yaml

   where <name> specifies the name for this configuration.

   After applying the configuration update, all the pods in sriov-network-operator namespace transition to the Running status.

4. To verify that the SR-IOV network device is configured, enter the following command. Replace <node_name> with the name of a node with the SR-IOV network device that you just configured.

   $ oc get sriovnetworknodestates -n openshift-sriov-network-operator <node_name> -o jsonpath='{.status.syncStatus}'

Configuring SR-IOV additional network

You can configure an additional network that uses SR-IOV hardware by creating an SriovNetwork object.

When you create an SriovNetwork object, the SR-IOV Network Operator automatically creates a NetworkAttachmentDefinition object.

Prerequisites

• Install the OpenShift CLI (oc).

• Log in as a user with cluster-admin privileges.

Procedure

1. Create the following SriovNetwork object, and then save the YAML in the <name>-sriov-network.yaml file. Replace <name> with a name for this additional network.

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
  name: <name> (1)
  namespace: openshift-sriov-network-operator (2)
spec:
  resourceName: <sriov_resource_name> (3)
  networkNamespace: <target_namespace> (4)
  vlan: <vlan> (5)
  spoofChk: "<spoof_check>" (6)
  linkState: <link_state> (7)
  maxTxRate: <max_tx_rate> (8)
  minTxRate: <min_rx_rate> (9)
  vlanQoS: <vlan_qos> (10)
  trust: "<trust_vf>" (11)
  capabilities: <capabilities> (12)

1. To create the object, enter the following command. Replace <name> with a name for this additional network.

   $ oc create -f <name>-sriov-network.yaml

2. Optional: To confirm that the NetworkAttachmentDefinition object associated with the SriovNetwork object that you created in the previous step exists, enter the following command. Replace <namespace> with the namespace you specified in the SriovNetwork object.

   $ oc get net-attach-def -n <namespace>

Connecting a virtual machine to an SR-IOV network

You can connect the virtual machine (VM) to the SR-IOV network by including the network details in the VM configuration.

Procedure

1. Include the SR-IOV network details in the spec.domain.devices.interfaces and spec.networks of the VM configuration:

   kind: VirtualMachine
   ...
   spec:
     domain:
       devices:
         interfaces:
         - name: <default> (1)
           masquerade: {} (2)
         - name: <nic1> (3)
           sriov: {}
     networks:
     - name: <default> (4)
       pod: {}
     - name: <nic1> (5)
       multus:
           networkName: <sriov-network> (6)
   ...

   1	A unique name for the interface that is connected to the pod network.
   2	The masquerade binding to the default pod network.
   3	A unique name for the SR-IOV interface.
   4	The name of the pod network interface. This must be the same as the interfaces.name that you defined earlier.
   5	The name of the SR-IOV interface. This must be the same as the interfaces.name that you defined earlier.
   6	The name of the SR-IOV network attachment definition.

2. Apply the virtual machine configuration:

   $ oc apply -f <vm-sriov.yaml> (1)

   1	The name of the virtual machine YAML file.