Optimizing data plane performance with the Intel FPGA PAC N3000 and Intel vRAN Dedicated Accelerator ACC100

Understanding Intel hardware accelerator cards for OKD

Hardware accelerator cards from Intel accelerate 4G/LTE and 5G Virtualized Radio Access Networks (vRAN) workloads. This in turn increases the overall compute capacity of a commercial, off-the-shelf platform.

Intel FPGA PAC N3000

The Intel FPGA PAC N3000 is a reference FPGA and uses 4G/LTE or 5G forward error correction (FEC) as an example workload that accelerates the 5G or 4G/LTE RAN layer 1 (L1) base station network function. Flash the Intel FPGA PAC N3000 card with 4G/LTE or 5G bitstreams to support vRAN workloads.

The Intel FPGA PAC N3000 is a full-duplex, 100 Gbps in-system, re-programmable acceleration card for multi-workload networking application acceleration.

When the Intel FPGA PAC N3000 is programmed with a 4G/LTE or 5G bitstream, it exposes the Single Root I/O Virtualization (SR-IOV) virtual function (VF) devices used to accelerate the FEC in the vRAN workload. To take advantage of this functionality for a cloud-native deployment, the physical function (PF) of the device must be bound to the pf-pci-stub driver to create several VFs. After the VFs are created, the VFs must be bound to a DPDK userspace driver (vfio) to allocate them to specific pods running the vRAN workload.

Intel FPGA PAC N3000 support on OKD depends on two Operators:

  • OpenNESS Operator for Intel FPGA PAC N3000 (Programming)

  • OpenNESS Operator for Wireless FEC Accelerators

vRAN Dedicated Accelerator ACC100

The vRAN Dedicated Accelerator ACC100, based on Intel’s eASIC technology is designed to offload and accelerate the computing-intensive process of forward error correction for 4G/LTE and 5G technology, freeing up processing power. Intel eASIC devices are structured ASICs, an intermediate technology between FPGAs and standard application-specific integrated circuits (ASICs).

Intel vRAN Dedicated Accelerator ACC100 support on OKD uses one Operator:

  • OpenNESS Operator for Wireless FEC Accelerators

Installing the OpenNESS Operator for Intel FPGA PAC N3000

The OpenNESS Operator for Intel FPGA PAC N3000 orchestrates and manages the resources or devices exposed by the Intel FPGA PAC N3000 card within the OKD cluster.

For vRAN use cases, the OpenNESS Operator for Intel FPGA PAC N3000 is used with the OpenNESS Operator for Wireless FEC Accelerators.

As a cluster administrator, you can install the OpenNESS Operator for Intel FPGA PAC N3000 by using the OKD CLI or the web console.

Installing the Operator by using the CLI

As a cluster administrator, you can install the Operator by using the CLI.

Prerequisites

  • A cluster installed on bare-metal hardware.

  • Install the OpenShift CLI (oc).

  • Log in as a user with cluster-admin privileges.

Procedure

  1. Create a namespace for the N3000 Operator by completing the following actions:

    1. Define the vran-acceleration-operators namespace by creating a file named n3000-namespace.yaml file as shown in the following example:

      1. apiVersion: v1
      2. kind: Namespace
      3. metadata:
      4.     name: vran-acceleration-operators
      5.     labels:
      6.         openshift.io/cluster-monitoring: "true"

    2. Create the namespace by running the following command:

      1. $ oc create -f n3000-namespace.yaml

  2. Install the N3000 Operator in the namespace you created in the previous step:

    1. Create the following OperatorGroup CR and save the YAML in the n3000-operatorgroup.yaml file:

      1. apiVersion: operators.coreos.com/v1
      2. kind: OperatorGroup
      3. metadata:
      4.     name: n3000-operators
      5.     namespace: vran-acceleration-operators
      6. spec:
      7.     targetNamespaces:
      8.     - vran-acceleration-operators

    2. Create the OperatorGroup CR by running the following command:

      1. $ oc create -f n3000-operatorgroup.yaml

    3. Run the following command to get the channel value required for the next step.

      1. $ oc get packagemanifest n3000 -n openshift-marketplace -o jsonpath='{.status.defaultChannel}'

      Example output

      1. stable

    4. Create the following Subscription CR and save the YAML in the n3000-sub.yaml file:

      1. apiVersion: operators.coreos.com/v1alpha1
      2. kind: Subscription
      3. metadata:
      4.     name: n3000-subscription
      5.     namespace: vran-acceleration-operators
      6. spec:
      7.     channel: "<channel>"        (1)
      8.     name: n3000
      9.     source: certified-operators (2)
      10.     sourceNamespace: openshift-marketplace
      1Specify the value for channel from the value obtained in the previous step for the .status.defaultChannel parameter.
      2You must specify the certified-operators value.

    5. Create the Subscription CR by running the following command:

      1. $ oc create -f n3000-sub.yaml

Verification

  • Verify the Operator is installed:

    1. $ oc get csv

    Example output

    1. NAME            DISPLAY                                         VERSION  REPLACES    PHASE
    2. n3000.v1.1.0    OpenNESS Operator for Intel® FPGA PAC N3000     1.1.0                Succeeded

    You have now successfully installed the Operator.

Installing the OpenNESS Operator for Intel FPGA PAC N3000 Operator by using the web console

As a cluster administrator, you can install the OpenNESS Operator for Intel FPGA PAC N3000 by using the web console.

You must create the Namespace and OperatorGroup CR as mentioned in the previous section.

Procedure

  1. Install the OpenNESS Operator for Intel FPGA PAC N3000 by using the OKD web console:

    1. In the OKD web console, click Operators → OperatorHub.

    2. Choose OpenNESS Operator for Intel FPGA PAC N3000 from the list of available Operators, and then click Install.

    3. On the Install Operator page, select All namespaces on the cluster. Then, click Install.

  2. Optional: Verify that the N3000 Operator is installed successfully:

    1. Switch to the Operators → Installed Operators page.

    2. Ensure that OpenNESS Operator for Intel FPGA PAC N3000 is listed in the vran-acceleration-operators project with a Status of InstallSucceeded.

      During installation, an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

      If the console does not indicate that the Operator is installed, perform the following troubleshooting steps:

      • Go to the Operators → Installed Operators page and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.

      • Go to the Workloads → Pods page and check the logs for pods in the vran-acceleration-operators project.

Programming the OpenNESS Operator for Intel FPGA PAC N3000

When the Intel FPGA PAC N3000 is programmed with a vRAN 5G bitstream, the hardware exposes the Intel FPGA PAC N3000 with a vRAN 5G bitstream. This bitstream exposes the Single Root I/O Virtualization (SR-IOV) virtual function (VF) devices used to accelerate the FEC in the vRAN workload.

As a cluster administrator, you can install the OpenNESS Operator for Intel FPGA PAC N3000 by using the OKD CLI or the web console.

Programming the N3000 with a vRAN bitstream

As a cluster administrator, you can program the Intel FPGA PAC N3000 with a vRAN 5G bitstream. This bitstream exposes the Single Root I/O Virtualization (SR-IOV) virtual function (VF) devices that are used to accelerate the forward error correction (FEC) in the vRAN workload.

The role of forward error correction (FEC) is to correct transmission errors, where certain bits in a message can be lost or garbled. Messages can be lost or garbled due to noise in the transmission media, interference, or low signal strength. Without FEC, a garbled message would have to be resent, adding to the network load and impacting both throughput and latency.

Prerequisites

  • Intel FPGA PAC N3000 card

  • Performance Addon Operator with RT kernel configuration

  • Node or nodes installed with the OpenNESS Operator for Intel FPGA PAC N3000

  • Log in as a user with cluster-admin privileges

    All the commands run in the vran-acceleration-operators namespace.

Procedure

  1. Change to the vran-acceleration-operators project:

    1. $ oc project vran-acceleration-operators

  2. Verify that the pods are running:

    1. $ oc get pods

    Example output

    1. NAME                                        READY       STATUS          RESTARTS    AGE
    2. fpga-driver-daemonset-8xz4c                 1/1         Running         0           15d
    3. fpgainfo-exporter-vhvdq                     1/1         Running         1           15d
    4. N3000-controller-manager-b68475c76-gcc6v    2/2         Running         1           15d
    5. N3000-daemonset-5k55l                       1/1         Running         1           15d
    6. N3000-discovery-blmjl                       1/1         Running         1           15d
    7. N3000-discovery-lblh7                       1/1         Running         1           15d

    The following section provides information on the installed pods:

    • fpga-driver-daemonset provides and loads the required Open Programmable Accelerator Engine (OPAE) drivers

    • fpgainfo-exporter provides N3000 telemetry data for Prometheus

    • N3000-controller-manager applies N3000Node CRs to the cluster and manages all the operand containers

    • N3000-daemonset is the main worker application. It monitors the changes in each node’s CR and acts on the changes. The logic implemented into this Daemon takes care of updating the cards’ FPGA user image and NIC firmware. It is also responsible for draining the nodes and taking them out of commission when required by the update.

    • N3000-discovery discovers N3000 Accelerator devices installed and labels worker nodes if devices are present

  3. Get all the nodes containing the Intel FPGA PAC N3000 card:

    1. $ oc get n3000node

    Example output

    1. NAME                       FLASH
    2. node1                      NotRequested

  4. Get information about the card on each node:

    1. $ oc get n3000node node1 -o yaml

    Example output

    1. status:
    2.   conditions:
    3.   - lastTransitionTime: "2020-12-15T17:09:26Z"
    4.     message: Inventory up to date
    5.     observedGeneration: 1
    6.     reason: NotRequested
    7.     status: "False"
    8.     type: Flashed
    9.   fortville:
    10.   - N3000PCI: 0000:1b:00.0
    11.     NICs:
    12.     - MAC: 64:4c:36:11:1b:a8
    13.       NVMVersion: 7.00 0x800052b0 0.0.0
    14.       PCIAddr: 0000:1a:00.0
    15.       name: Ethernet Controller XXV710 Intel(R) FPGA Programmable Acceleration Card N3000 for Networking
    16.     - MAC: 64:4c:36:11:1b:a9
    17.       NVMVersion: 7.00 0x800052b0 0.0.0
    18.       PCIAddr: 0000:1a:00.1
    19.       name: Ethernet Controller XXV710 Intel(R) FPGA Programmable Acceleration Card N3000 for Networking
    20.     - MAC: 64:4c:36:11:1b:ac
    21.       NVMVersion: 7.00 0x800052b0 0.0.0
    22.       PCIAddr: 0000:1c:00.0
    23.       name: Ethernet Controller XXV710 Intel(R) FPGA Programmable Acceleration Card N3000 for Networking
    24.     - MAC: 64:4c:36:11:1b:ad
    25.       NVMVersion: 7.00 0x800052b0 0.0.0
    26.       PCIAddr: 0000:1c:00.1
    27.       name: Ethernet Controller XXV710 Intel(R) FPGA Programmable Acceleration Card N3000 for Networking
    28.   fpga:
    29.   - PCIAddr: 0000:1b:00.0 (1)
    30.     bitstreamId: "0x23000410010310" (2)
    31.     bitstreamVersion: 0.2.3
    32.     deviceId: "0x0b30"
    1The PCIAddr field indicates the PCI address of the card.
    2The bitstreamId field indicates the bitstream that is currently stored in flash.

  5. Save the current bitstreamId, PCIAddr, the name, and the deviceId without “0x” padding.

    1. $ oc get n3000node -o json

  6. Update the user bitstream of the Intel FPGA PAC N3000 card:

    1. Define the N3000 cluster resource to program by creating a file named n3000-cluster.yaml as shown in the following example:

      1. apiVersion: fpga.intel.com/v1
      2. kind: N3000Cluster
      3. metadata:
      4.     name: n3000 (1)
      5.     namespace: vran-acceleration-operators
      6. spec:
      7.     nodes:
      8.       - nodeName: "node1" (2)
      9.         fpga:
      10.           - userImageURL: "http://10.10.10.122:8000/pkg/20ww27.5-2x2x25G-5GLDPC-v1.6.1-3.0.0_unsigned.bin" (3)
      11.             PCIAddr: "0000:1b:00.0" (4)
      12.             checksum: "0b0a87b974d35ea16023ceb57f7d5d9c" (5)
      1Specify the name. The name must be n3000.
      2Specify the node to program.
      3Specify the URL for the user bitstream. This bitstream file must be accessible on an HTTP or HTTPS server.
      4Specify the PCI address of the card to program.
      5Specify the MD5 checksum of the bitstream that is specified in the userImageURL field.

      The N3000 daemon updates the FPGA user bitstream using the Open Programmable Acceleration Engine (OPAE) tools and resets the PCI device. The update of the FPGA user bitstream can require up to 40 minutes per card. For programming cards on multiple nodes, the programming happens one node at a time.

    2. Apply the update to begin programming the card with the bitstream:

      1. $ oc apply -f n3000-cluster.yaml

      The N3000 daemon starts programming the bitstream after the appropriate 5G FEC user bitstream has been provisioned, such as 20ww27.5-2x2x25G-5GLDPC-v1.6.1-3.0.0_unsigned.bin in this example, and after the CR has been created.

    3. Check the status:

      1. oc get n3000node

      Example output

      1. NAME             FLASH
      2. node1            InProgress

  7. Check the logs:

    1. Determine the pod name of the N3000 daemon:

      1. $ oc get pod -o wide | grep n3000-daemonset | grep node1

      Example output

      1. n3000-daemonset-5k55l              1/1     Running   0          15d

    2. View the logs:

      1. $ oc logs n3000-daemonset-5k55l

      Example output

      1. ...
      2. {"level":"info","ts":1608054338.8866854,"logger":"daemon.drainhelper.cordonAndDrain()","msg":"node drained"}
      3. {"level":"info","ts":1608054338.8867319,"logger":"daemon.drainhelper.Run()","msg":"worker function - start"}
      4. {"level":"info","ts":1608054338.9003832,"logger":"daemon.fpgaManager.ProgramFPGAs","msg":"Start program","PCIAddr":"0000:1b:00.0"}
      5. {"level":"info","ts":1608054338.9004142,"logger":"daemon.fpgaManager.ProgramFPGA","msg":"Starting","pci":"0000:1b:00.0"}
      6. {"level":"info","ts":1608056309.9367146,"logger":"daemon.fpgaManager.ProgramFPGA","msg":"Program FPGA completed, start new power cycle N3000 ...","pci":"0000:1b:00.0"}
      7. {"level":"info","ts":1608056333.3528838,"logger":"daemon.drainhelper.Run()","msg":"worker function - end","performUncordon":true}
      8. ...

      The log file indicates the following flow of events:

      • The bitstream is downloaded and validated.

      • The node is drained and no workload is able to run during this time.

      • Flashing is started:

        • The bitstream is flashed into the card.

        • The bitstream is applied.

      • After flashing is complete the PCI device or devices on the node or nodes are reloaded. The OpenNESS SR-IOV Operator for Wireless FEC Accelerators is now able to find the new flashed device or devices.

Verification

  1. Verify the status after the FPGA user bitstream update is complete:

    1. oc get n3000node

    Example output

    1. NAME             FLASH
    2. node1            Succeeded

  2. Verify that the bitstream ID of the card has changed:

    1. oc get n3000node node1 -o yaml

    Example output

    1. status:
    2.       conditions:
    3.           - lastTransitionTime: "2020-12-15T18:18:53Z"
    4.             message: Flashed successfully (1)
    5.             observedGeneration: 2
    6.             reason: Succeeded
    7.             status: "True"
    8.             type: Flashed
    9.       fortville:
    10.       - N3000PCI: 0000:1b:00.0
    11.         NICs:
    12.         - MAC: 64:4c:36:11:1b:a8
    13.           NVMVersion: 7.00 0x800052b0 0.0.0
    14.           PCIAddr: 0000:1a:00.0
    15.           name: Ethernet Controller XXV710 Intel(R) FPGA Programmable Acceleration Card N3000 for Networking
    16.         - MAC: 64:4c:36:11:1b:a9
    17.           NVMVersion: 7.00 0x800052b0 0.0.0
    18.           PCIAddr: 0000:1a:00.1
    19.           name: Ethernet Controller XXV710 Intel(R) FPGA Programmable Acceleration Card N3000 for Networking
    20.         - MAC: 64:4c:36:11:1b:ac
    21.           NVMVersion: 7.00 0x800052b0 0.0.0
    22.           PCIAddr: 0000:1c:00.0
    23.           name: Ethernet Controller XXV710 Intel(R) FPGA Programmable Acceleration Card N3000 for Networking
    24.         - MAC: 64:4c:36:11:1b:ad
    25.           NVMVersion: 7.00 0x800052b0 0.0.0
    26.           PCIAddr: 0000:1c:00.1
    27.           name: Ethernet Controller XXV710 Intel(R) FPGA Programmable Acceleration Card N3000 for Networking
    28.       fpga:
    29.       - PCIAddr: 0000:1b:00.0 (2)
    30.         bitstreamId: "0x2315842A010601" (3)
    31.         bitstreamVersion: 0.2.3
    32.         deviceId: "0x0b30" (4)
    1The message field indicates the device is successfully flashed.
    2The PCIAddr field indicates the PCI address of the card.
    3The bitstreamId field indicates the updated bitstream ID.
    4The deviceID field indicates that device ID of the bitstream inside the card exposed to the system.

  3. Check the FEC PCI devices on the node:

    1. Verify the node configuration is applied correctly:

      1. $ oc debug node/node1

      Expected output

      1. Starting pod/<node-name>-debug ...
      2. To use host binaries, run `chroot /host`
      4. Pod IP: <ip-address>
      5. If you don't see a command prompt, try pressing enter.
      7. sh-4.4#

    2. Verify that you can use the node file system:

      1. sh-4.4# chroot /host

      Expected output

      1. sh-4.4#

    3. List the PCI devices associated with the accelerator on your system:

      1. $ lspci | grep accelerators

      Expected output

      1. 1b:00.0 Processing accelerators: Intel Corporation Device 0b30
      2. 1d:00.0 Processing accelerators: Intel Corporation Device 0d8f (rev 01)

      Devices belonging to the FPGA are reported in the output. Device ID 0b30 is the RSU interface used to program the card, and the 0d8f is a physical function of the newly programmed 5G device.

Installing the OpenNESS SR-IOV Operator for Wireless FEC Accelerators

The role of the OpenNESS SR-IOV Operator for Wireless FEC Accelerators is to orchestrate and manage the devices exposed by a range of Intel vRAN FEC acceleration hardware within the OKD cluster.

One of the most compute-intensive 4G/LTE and 5G workloads is RAN layer 1 (L1) forward error correction (FEC). FEC resolves data transmission errors over unreliable or noisy communication channels. FEC technology detects and corrects a limited number of errors in 4G/LTE or 5G data without the need for retransmission.

The FEC devices are provided by the Intel FPGA PAC N3000 and the Intel vRAN Dedicated Accelerator ACC100 for the vRAN use case.

The Intel FPGA PAC N3000 FPGA requires flashing with an 4G/LTE or 5G bitstream.

The OpenNESS SR-IOV Operator for Wireless FEC Accelerators provides functionality to create virtual functions (VFs) for the FEC device, binds them to appropriate drivers, and configures the VFs queues for functionality in 4G/LTE or 5G deployment.

As a cluster administrator, you can install the OpenNESS SR-IOV Operator for Wireless FEC Accelerators by using the OKD CLI or the web console.

Installing the OpenNESS SR-IOV Operator for Wireless FEC Accelerators by using the CLI

As a cluster administrator, you can install the OpenNESS SR-IOV Operator for Wireless FEC Accelerators by using the CLI.

Prerequisites

  • A cluster installed on bare-metal hardware.

  • Install the OpenShift CLI (oc).

  • Log in as a user with cluster-admin privileges.

Procedure

  1. Create a namespace for the OpenNESS SR-IOV Operator for Wireless FEC Accelerators by completing the following actions:

    1. Define the vran-acceleration-operators namespace by creating a file named sriov-namespace.yaml as shown in the following example:

      1. apiVersion: v1
      2. kind: Namespace
      3. metadata:
      4.     name: vran-acceleration-operators
      5.     labels:
      6.        openshift.io/cluster-monitoring: "true"

    2. Create the namespace by running the following command:

      1. $ oc create -f sriov-namespace.yaml

  2. Install the OpenNESS SR-IOV Operator for Wireless FEC Accelerators in the namespace you created in the previous step by creating the following objects:

    1. Create the following OperatorGroup CR and save the YAML in the sriov-operatorgroup.yaml file:

      1. apiVersion: operators.coreos.com/v1
      2. kind: OperatorGroup
      3. metadata:
      4.     name: vran-operators
      5.     namespace: vran-acceleration-operators
      6. spec:
      7.     targetNamespaces:
      8.       - vran-acceleration-operators

    2. Create the OperatorGroup CR by running the following command:

      1. $ oc create -f sriov-operatorgroup.yaml

    3. Run the following command to get the channel value required for the next step.

      1. $ oc get packagemanifest sriov-fec -n openshift-marketplace -o jsonpath='{.status.defaultChannel}'

      Example output

      1. stable

    4. Create the following Subscription CR and save the YAML in the sriov-sub.yaml file:

      1. apiVersion: operators.coreos.com/v1alpha1
      2. kind: Subscription
      3. metadata:
      4.     name: sriov-fec-subscription
      5.     namespace: vran-acceleration-operators
      6. spec:
      7.     channel: "<channel>" (1)
      8.     name: sriov-fec
      9.     source: certified-operators (2)
      10.     sourceNamespace: openshift-marketplace
      1Specify the value for channel from the value obtained in the previous step for the .status.defaultChannel parameter.
      2You must specify the certified-operators value.

    5. Create the Subscription CR by running the following command:

      1. $ oc create -f sriov-sub.yaml

Verification

  • Verify that the Operator is installed:

    1. $ oc get csv -n vran-acceleration-operators -o custom-columns=Name:.metadata.name,Phase:.status.phase

    Example output

    1. Name                                        Phase
    2. sriov-fec.v1.1.0                            Succeeded

Installing the OpenNESS SR-IOV Operator for Wireless FEC Accelerators by using the web console

As a cluster administrator, you can install the OpenNESS SR-IOV Operator for Wireless FEC Accelerators by using the web console.

You must create the Namespace and OperatorGroup CR as mentioned in the previous section.

Procedure

  1. Install the OpenNESS SR-IOV Operator for Wireless FEC Accelerators by using the OKD web console:

    1. In the OKD web console, click Operators → OperatorHub.

    2. Choose OpenNESS SR-IOV Operator for Wireless FEC Accelerators from the list of available Operators, and then click Install.

    3. On the Install Operator page, select All namespaces on the cluster. Then, click Install.

  2. Optional: Verify that the SRIOV-FEC Operator is installed successfully:

    1. Switch to the Operators → Installed Operators page.

    2. Ensure that OpenNESS SR-IOV Operator for Wireless FEC Accelerators is listed in the vran-acceleration-operators project with a Status of InstallSucceeded.

      During installation an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

      If the console does not indicate that the Operator is installed, perform the following troubleshooting steps:

      • Go to the Operators → Installed Operators page and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.

      • Go to the Workloads → Pods page and check the logs for pods in the vran-acceleration-operators project.

Configuring the SR-IOV-FEC Operator for Intel FPGA PAC N3000

This section describes how to program the SR-IOV-FEC Operator for Intel FPGA PAC N3000. The SR-IOV-FEC Operator handles the management of the forward error correction (FEC) devices that are used to accelerate the FEC process in vRAN L1 applications.

Configuring the SR-IOV-FEC Operator involves:

  • Creating the desired virtual functions (VFs) for the FEC device

  • Binding the VFs to the appropriate drivers

  • Configuring the VF queues for desired functionality in a 4G or 5G deployment

The role of forward error correction (FEC) is to correct transmission errors, where certain bits in a message can be lost or garbled. Messages can be lost or garbled due to noise in the transmission media, interference, or low signal strength. Without FEC, a garbled message would have to be resent, adding to the network load and impacting throughput and latency.

Prerequisites

  • Intel FPGA PAC N3000 card

  • Node or nodes installed with the OpenNESS Operator for Intel FPGA PAC N3000 (Programming)

  • Node or nodes installed with the OpenNESS Operator for Wireless FEC Accelerators

  • RT kernel configured with Performance Addon Operator

Procedure

  1. Change to the vran-acceleration-operators project:

    1. $ oc project vran-acceleration-operators

  2. Verify that the SR-IOV-FEC Operator is installed:

    1. $ oc get csv -o custom-columns=Name:.metadata.name,Phase:.status.phase

    Example output

    1. Name                                        Phase
    2. sriov-fec.v1.1.0                            Succeeded
    3. n3000.v1.1.0                                Succeeded

  3. Verify that the N3000 and sriov-fec pods are running:

    1. $  oc get pods

    Example output

    1. NAME                                            READY       STATUS      RESTARTS    AGE
    2. fpga-driver-daemonset-8xz4c                     1/1         Running     0           15d
    3. fpgainfo-exporter-vhvdq                         1/1         Running     1           15d
    4. N3000-controller-manager-b68475c76-gcc6v        2/2         Running     1           15d
    5. N3000-daemonset-5k55l                           1/1         Running     1           15d
    6. N3000-discovery-blmjl                           1/1         Running     1           15d
    7. N3000-discovery-lblh7                           1/1         Running     1           15d
    8. sriov-device-plugin-j5jlv                       1/1         Running     1           15d
    9. sriov-fec-controller-manager-85b6b8f4d4-gd2qg   1/1         Running     1           15d
    10. sriov-fec-daemonset-kqqs6                       1/1         Running     1           15d

    The following section provides information on the installed pods:

    • fpga-driver-daemonset provides and loads the required Open Programmable Accelerator Engine (OPAE) drivers

    • fpgainfo-exporter provides N3000 telemetry data for Prometheus

    • N3000-controller-manager applies N3000Node CRs to the cluster and manages all the operand containers

    • N3000-daemonset is the main worker application

    • N3000-discovery discovers N3000 Accelerator devices installed and labels worker nodes if devices are present

    • sriov-device-plugin expose the FEC virtual functions as resources under the node

    • sriov-fec-controller-manager applies CR to the node and maintains the operands containers

    • sriov-fec-daemonset is responsible for:

      • Discovering the SRIOV NICs on each node.

      • Syncing the status of the custom resource (CR) defined in step 6.

      • Taking the spec of the CR as input and configuring the discovered NICs.

  1. Retrieve all the nodes containing one of the supported vRAN FEC accelerator devices:

    1. $ oc get sriovfecnodeconfig

    Example output

    1. NAME             CONFIGURED
    2. node1            Succeeded

  2. Find the physical function (PF) of the SR-IOV FEC accelerator device to configure:

    1. $ oc get sriovfecnodeconfig node1 -o yaml

    Example output

    1. status:
    2.     conditions:
    3.     - lastTransitionTime: "2021-03-19T17:19:37Z"
    4.       message: Configured successfully
    5.       observedGeneration: 1
    6.       reason: ConfigurationSucceeded
    7.       status: "True"
    8.       type: Configured
    9.     inventory:
    10.         sriovAccelerators:
    11.         - deviceID: 0d5c
    12.           driver: ""
    13.           maxVirtualFunctions: 16
    14.           pciAddress: 0000.1d.00.0 (1)
    15.           vendorID: "8086"
    16.           virtualFunctions: [] (2)
    1This field indicates the PCI Address of the card.
    2This field shows that the virtual functions are empty.

  3. Configure the FEC device with the desired setting.

    1. Create the following custom resource (CR) and save the YAML in the sriovfec_n3000_cr.yaml file:

      1. apiVersion: sriovfec.intel.com/v1
      2. kind: SriovFecClusterConfig
      3. metadata:
      4.     name: config
      5.     namespace: vran-acceleration-operators
      6. spec:
      7.   nodes:
      8.     - nodeName: node1 (1)
      9.       physicalFunctions:
      10.         - pciAddress: 0000:1d:00.0 (2)
      11.           pfDriver: pci-pf-stub
      12.           vfDriver: vfio-pci
      13.           vfAmount: 2 (3)
      14.           bbDevConfig:
      15.             n3000:
      16.               # Network Type: either "FPGA_5GNR" or "FPGA_LTE"
      17.               networkType: "FPGA_5GNR"
      18.               pfMode: false
      19.               flrTimeout: 610
      20.               downlink:
      21.                 bandwidth: 3
      22.                 loadBalance: 128
      23.                 queues: (4)
      24.                   vf0: 16
      25.                   vf1: 16
      26.                   vf2: 0
      27.                   vf3: 0
      28.                   vf4: 0
      29.                   vf5: 0
      30.                   vf6: 0
      31.                   vf7: 0
      32.               uplink:
      33.                 bandwidth: 3
      34.                 loadBalance: 128
      35.                 queues: (5)
      36.                   vf0: 16
      37.                   vf1: 16
      38.                   vf2: 0
      39.                   vf3: 0
      40.                   vf4: 0
      41.                   vf5: 0
      42.                   vf6: 0
      43.                   vf7: 0
      1Specify the node name.
      2Specify the PCI Address of the card on which the SR-IOV-FEC Operator will be installed.
      3Specify the number of virtual functions. Create two virtual functions.
      4On vf0 create one queue with 16 buses (downlink and uplink).
      5On vf1 create one queue with 16 buses (downlink and uplink).

      For Intel PAC N3000 for vRAN Acceleration the user can create up to 8 VF devices. Each FEC PF device provides a total of 64 queues to be configured, 32 queues for uplink and 32 queues for downlink. The queues would be typically distributed evenly across the VFs.

    2. Apply the CR:

      1. $ oc apply -f sriovfec_n3000_cr.yaml

      After applying the CR, the SR-IOV FEC daemon starts configuring the FEC device.

Verification

  1. Check the status:

    1. $ oc get sriovfecclusterconfig config -o yaml

    Example output

    1. status:
    2.     conditions:
    3.     - lastTransitionTime: "2020-12-15T17:19:37Z"
    4.       message: Configured successfully
    5.       observedGeneration: 1
    6.       reason: ConfigurationSucceeded
    7.       status: "True"
    8.       type: Configured
    9.     inventory:
    10.       sriovAccelerators:
    11.       - deviceID: 0d8f
    12.         driver: pci-pf-stub
    13.         maxVirtualFunctions: 8
    14.         pciAddress: 0000:1d:00.0
    15.         vendorID: "8086"
    16.         virtualFunctions:
    17.         - deviceID: 0d90
    18.           driver: vfio-pci
    19.           pciAddress: 0000:1d:00.1
    20.         - deviceID: 0d90
    21.           driver: vfio-pci
    22.           pciAddress: 0000:1d:00.2

  2. Check the logs:

    1. Determine the name of the SR-IOV daemon pod:

      1. $ oc get pod | grep sriov-fec-daemonset

      Example output

      1. sriov-fec-daemonset-kqqs6                      1/1     Running   0          19h

    2. View the logs:

      1. $ oc logs sriov-fec-daemonset-kqqs6

      Example output

      1. 2020-12-16T12:46:47.720Z        INFO    daemon.NodeConfigurator.applyConfig     configuring PF  {"requestedConfig": {"pciAddress":"0000:1d:00.0","pfDriver":"pci-pf-stub","vfDriver":"vfio-pci","vfAmount":2,"bbDevConfig":{"n3000":{
      2. "networkType":"FPGA_5GNR","pfMode":false,"flrTimeout":610,"downlink":{"bandwidth":3,"loadBalance":128,"queues":{"vf0":16,"vf1":16}},"uplink":{"bandwidth":3,"loadBalance":128,"queues":{"vf0":16,"vf1":16}}}}}}
      3. 2020-12-16T12:46:47.720Z        INFO    daemon.NodeConfigurator.loadModule      executing command       {"cmd": "/usr/sbin/chroot /host/ modprobe pci-pf-stub"}
      4. 2020-12-16T12:46:47.724Z        INFO    daemon.NodeConfigurator.loadModule      commands output {"output": ""}
      5. 2020-12-16T12:46:47.724Z        INFO    daemon.NodeConfigurator.loadModule      executing command       {"cmd": "/usr/sbin/chroot /host/ modprobe vfio-pci"}
      6. 2020-12-16T12:46:47.727Z        INFO    daemon.NodeConfigurator.loadModule      commands output {"output": ""}
      7. 2020-12-16T12:46:47.727Z        INFO    daemon.NodeConfigurator device's driver_override path   {"path": "/sys/bus/pci/devices/0000:1d:00.0/driver_override"}
      8. 2020-12-16T12:46:47.727Z        INFO    daemon.NodeConfigurator driver bind path        {"path": "/sys/bus/pci/drivers/pci-pf-stub/bind"}
      9. 2020-12-16T12:46:47.998Z        INFO    daemon.NodeConfigurator device's driver_override path   {"path": "/sys/bus/pci/devices/0000:1d:00.1/driver_override"}
      10. 2020-12-16T12:46:47.998Z        INFO    daemon.NodeConfigurator driver bind path        {"path": "/sys/bus/pci/drivers/vfio-pci/bind"}
      11. 2020-12-16T12:46:47.998Z        INFO    daemon.NodeConfigurator device's driver_override path   {"path": "/sys/bus/pci/devices/0000:1d:00.2/driver_override"}
      12. 2020-12-16T12:46:47.998Z        INFO    daemon.NodeConfigurator driver bind path        {"path": "/sys/bus/pci/drivers/vfio-pci/bind"}
      13. 2020-12-16T12:46:47.999Z        INFO    daemon.NodeConfigurator.applyConfig     executing command       {"cmd": "/sriov_workdir/pf_bb_config FPGA_5GNR -c /sriov_artifacts/0000:1d:00.0.ini -p 0000:1d:00.0"}
      14. 2020-12-16T12:46:48.017Z        INFO    daemon.NodeConfigurator.applyConfig     commands output {"output": "ERROR: Section (FLR) or name (flr_time_out) is not valid.
      15. FEC FPGA RTL v3.0
      16. UL.DL Weights = 3.3
      17. UL.DL Load Balance = 1
      18. 28.128
      19. Queue-PF/VF Mapping Table = READY
      20. Ring Descriptor Size = 256 bytes
      22. --------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
      23.         |  PF | VF0 | VF1 | VF2 | VF3 | VF4 | VF5 | VF6 | VF7 |
      24. --------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
      25. UL-Q'00 |     |  X  |     |     |     |     |     |     |     |
      26. UL-Q'01 |     |  X  |     |     |     |     |     |     |     |
      27. UL-Q'02 |     |  X  |     |     |     |     |     |     |     |
      28. UL-Q'03 |     |  X  |     |     |     |     |     |     |     |
      29. UL-Q'04 |     |  X  |     |     |     |     |     |     |     |
      30. UL-Q'05 |     |  X  |     |     |     |     |     |     |     |
      31. UL-Q'06 |     |  X  |     |     |     |     |     |     |     |
      32. UL-Q'07 |     |  X  |     |     |     |     |     |     |     |
      33. UL-Q'08 |     |  X  |     |     |     |     |     |     |     |
      34. UL-Q'09 |     |  X  |     |     |     |     |     |     |     |
      35. UL-Q'10 |     |  X  |     |     |     |     |     |     |     |
      36. UL-Q'11 |     |  X  |     |     |     |     |     |     |     |
      37. UL-Q'12 |     |  X  |     |     |     |     |     |     |     |
      38. UL-Q'13 |     |  X  |     |     |     |     |     |     |     |
      39. UL-Q'14 |     |  X  |     |     |     |     |     |     |     |
      40. UL-Q'15 |     |  X  |     |     |     |     |     |     |     |
      41. UL-Q'16 |     |     |  X  |     |     |     |     |     |     |
      42. UL-Q'17 |     |     |  X  |     |     |     |     |     |     |
      43. UL-Q'18 |     |     |  X  |     |     |     |     |     |     |
      44. UL-Q'19 |     |     |  X  |     |     |     |     |     |     |
      45. UL-Q'20 |     |     |  X  |     |     |     |     |     |     |
      46. UL-Q'21 |     |     |  X  |     |     |     |     |     |     |
      47. UL-Q'22 |     |     |  X  |     |     |     |     |     |     |
      48. UL-Q'23 |     |     |  X  |     |     |     |     |     |     |
      49. UL-Q'24 |     |     |  X  |     |     |     |     |     |     |
      50. UL-Q'25 |     |     |  X  |     |     |     |     |     |     |
      51. UL-Q'26 |     |     |  X  |     |     |     |     |     |     |
      52. UL-Q'27 |     |     |  X  |     |     |     |     |     |     |
      53. UL-Q'28 |     |     |  X  |     |     |     |     |     |     |
      54. UL-Q'29 |     |     |  X  |     |     |     |     |     |     |
      55. UL-Q'30 |     |     |  X  |     |     |     |     |     |     |
      56. UL-Q'31 |     |     |  X  |     |     |     |     |     |     |
      57. DL-Q'32 |     |  X  |     |     |     |     |     |     |     |
      58. DL-Q'33 |     |  X  |     |     |     |     |     |     |     |
      59. DL-Q'34 |     |  X  |     |     |     |     |     |     |     |
      60. DL-Q'35 |     |  X  |     |     |     |     |     |     |     |
      61. DL-Q'36 |     |  X  |     |     |     |     |     |     |     |
      62. DL-Q'37 |     |  X  |     |     |     |     |     |     |     |
      63. DL-Q'38 |     |  X  |     |     |     |     |     |     |     |
      64. DL-Q'39 |     |  X  |     |     |     |     |     |     |     |
      65. DL-Q'40 |     |  X  |     |     |     |     |     |     |     |
      66. DL-Q'41 |     |  X  |     |     |     |     |     |     |     |
      67. DL-Q'42 |     |  X  |     |     |     |     |     |     |     |
      68. DL-Q'43 |     |  X  |     |     |     |     |     |     |     |
      69. DL-Q'44 |     |  X  |     |     |     |     |     |     |     |
      70. DL-Q'45 |     |  X  |     |     |     |     |     |     |     |
      71. DL-Q'46 |     |  X  |     |     |     |     |     |     |     |
      72. DL-Q'47 |     |  X  |     |     |     |     |     |     |     |
      73. DL-Q'48 |     |     |  X  |     |     |     |     |     |     |
      74. DL-Q'49 |     |     |  X  |     |     |     |     |     |     |
      75. DL-Q'50 |     |     |  X  |     |     |     |     |     |     |
      76. DL-Q'51 |     |     |  X  |     |     |     |     |     |     |
      77. DL-Q'52 |     |     |  X  |     |     |     |     |     |     |
      78. DL-Q'53 |     |     |  X  |     |     |     |     |     |     |
      79. DL-Q'54 |     |     |  X  |     |     |     |     |     |     |
      80. DL-Q'55 |     |     |  X  |     |     |     |     |     |     |
      81. DL-Q'56 |     |     |  X  |     |     |     |     |     |     |
      82. DL-Q'57 |     |     |  X  |     |     |     |     |     |     |
      83. DL-Q'58 |     |     |  X  |     |     |     |     |     |     |
      84. DL-Q'59 |     |     |  X  |     |     |     |     |     |     |
      85. DL-Q'60 |     |     |  X  |     |     |     |     |     |     |
      86. DL-Q'61 |     |     |  X  |     |     |     |     |     |     |
      87. DL-Q'62 |     |     |  X  |     |     |     |     |     |     |
      88. DL-Q'63 |     |     |  X  |     |     |     |     |     |     |
      89. --------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
      91. Mode of operation = VF-mode
      92. FPGA_5GNR PF [0000:1d:00.0] configuration complete!"}
      93. 2020-12-16T12:46:48.017Z        INFO    daemon.NodeConfigurator.enableMasterBus executing command       {"cmd": "/usr/sbin/chroot /host/ setpci -v -s 0000:1d:00.0 COMMAND"}
      94. 2020-12-16T12:46:48.037Z        INFO    daemon.NodeConfigurator.enableMasterBus commands output {"output": "0000:1d:00.0 @04 = 0102\n"}
      95. 2020-12-16T12:46:48.037Z        INFO    daemon.NodeConfigurator.enableMasterBus executing command       {"cmd": "/usr/sbin/chroot /host/ setpci -v -s 0000:1d:00.0 COMMAND=0106"}
      96. 2020-12-16T12:46:48.054Z        INFO    daemon.NodeConfigurator.enableMasterBus commands output {"output": "0000:1d:00.0 @04 0106\n"}
      97. 2020-12-16T12:46:48.054Z        INFO    daemon.NodeConfigurator.enableMasterBus MasterBus set   {"pci": "0000:1d:00.0", "output": "0000:1d:00.0 @04 0106\n"}
      98. 2020-12-16T12:46:48.160Z        INFO    daemon.drainhelper.Run()        worker function - end   {"performUncordon": true}

  3. Check the FEC configuration of the card:

    1. $ oc get sriovfecnodeconfig node1 -o yaml

    Example output

    1. status:
    2.     conditions:
    3.     - lastTransitionTime: "2020-12-15T17:19:37Z"
    4.       message: Configured successfully
    5.       observedGeneration: 1
    6.       reason: ConfigurationSucceeded
    7.       status: "True"
    8.       type: Configured
    9.     inventory:
    10.       sriovAccelerators:
    11.       - deviceID: 0d8f (1)
    12.         driver: pci-pf-stub
    13.         maxVirtualFunctions: 8
    14.         pciAddress: 0000:1d:00.0
    15.         vendorID: "8086"
    16.       virtualFunctions:
    17.       - deviceID: 0d90 (2)
    18.         driver: vfio-pci
    19.         pciAddress: 0000:1d:00.1
    20.       - deviceID: 0d90
    21.         driver: vfio-pci
    22.         pciAddress: 0000:1d:00.2
    10d8f is the deviceID physical function of the FEC device.
    20d90 is the deviceID virtual function of the FEC device.

Configuring the SR-IOV-FEC Operator for the Intel vRAN Dedicated Accelerator ACC100

Programming the Intel vRAN Dedicated Accelerator ACC100 exposes the Single Root I/O Virtualization (SRIOV) virtual function (VF) devices that are then used to accelerate the FEC in the vRAN workload. The Intel vRAN Dedicated Accelerator ACC100 accelerates 4G and 5G Virtualized Radio Access Networks (vRAN) workloads. This in turn increases the overall compute capacity of a commercial, off-the-shelf platform. This device is also known as Mount Bryce.

The SR-IOV-FEC Operator handles the management of the forward error correction (FEC) devices that are used to accelerate the FEC process in vRAN L1 applications.

Configuring the SR-IOV-FEC Operator involves:

  • Creating the virtual functions (VFs) for the FEC device

  • Binding the VFs to the appropriate drivers

  • Configuring the VF queues for desired functionality in a 4G or 5G deployment

The role of forward error correction (FEC) is to correct transmission errors, where certain bits in a message can be lost or garbled. Messages can be lost or garbled due to noise in the transmission media, interference, or low signal strength. Without FEC, a garbled message would have to be resent, adding to the network load and impacting throughput and latency.

Prerequisites

  • Intel FPGA ACC100 5G/4G card

  • Node or nodes installed with the OpenNESS Operator for Wireless FEC Accelerators

  • Enable global SR-IOV and VT-d settings in the BIOS for the node

  • RT kernel configured with Performance Addon Operator

  • Log in as a user with cluster-admin privileges

Procedure

  1. Change to the vran-acceleration-operators project:

    1. $ oc project vran-acceleration-operators

  2. Verify that the SR-IOV-FEC Operator is installed:

    1. $ oc get csv -o custom-columns=Name:.metadata.name,Phase:.status.phase

    Example output

    1. Name                                        Phase
    2. sriov-fec.v1.1.0                            Succeeded

  3. Verify that the sriov-fec pods are running:

    1. $  oc get pods

    Example output

    1. NAME                                            READY       STATUS      RESTARTS    AGE
    2. sriov-device-plugin-j5jlv                       1/1         Running     1           15d
    3. sriov-fec-controller-manager-85b6b8f4d4-gd2qg   1/1         Running     1           15d
    4. sriov-fec-daemonset-kqqs6                       1/1         Running     1           15d

    • sriov-device-plugin expose the FEC virtual functions as resources under the node

    • sriov-fec-controller-manager applies CR to the node and maintains the operands containers

    • sriov-fec-daemonset is responsible for:

      • Discovering the SRIOV NICs on each node.

      • Syncing the status of the custom resource (CR) defined in step 6.

      • Taking the spec of the CR as input and configuring the discovered NICs.

  1. Retrieve all the nodes containing one of the supported vRAN FEC accelerator devices:

    1. $ oc get sriovfecnodeconfig

    Example output

    1. NAME             CONFIGURED
    2. node1            Succeeded

  2. Find the physical function (PF) of the SR-IOV FEC accelerator device to configure:

    1. $ oc get sriovfecnodeconfig node1 -o yaml

    Example output

    1. status:
    2.     conditions:
    3.     - lastTransitionTime: "2021-03-19T17:19:37Z"
    4.       message: Configured successfully
    5.       observedGeneration: 1
    6.       reason: ConfigurationSucceeded
    7.       status: "True"
    8.       type: Configured
    9.     inventory:
    10.        sriovAccelerators:
    11.        - deviceID: 0d5c
    12.          driver: ""
    13.          maxVirtualFunctions: 16
    14.          pciAddress: 0000:af:00.0 (1)
    15.          vendorID: "8086"
    16.          virtualFunctions: [] (2)
    1This field indicates the PCI address of the card.
    2This field shows that the virtual functions are empty.

  3. Configure the number of virtual functions and queue groups on the FEC device:

    1. Create the following custom resource (CR) and save the YAML in the sriovfec_acc100cr.yaml file:

      This example configures the ACC100 8/8 queue groups for 5G, 4 queue groups for Uplink, and another 4 queue groups for Downlink.

      1. apiVersion: sriovfec.intel.com/v1
      2. kind: SriovFecClusterConfig
      3. metadata:
      4.   name: config (1)
      5. spec:
      6.   nodes:
      7.    - nodeName: node1 (2)
      8.      physicalFunctions:
      9.        - pciAddress: 0000:af:00.0 (3)
      10.          pfDriver: "pci-pf-stub"
      11.          vfDriver: "vfio-pci"
      12.          vfAmount: 16 (4)
      13.          bbDevConfig:
      14.            acc100:
      15.              # Programming mode: 0 = VF Programming, 1 = PF Programming
      16.              pfMode: false
      17.              numVfBundles: 16
      18.              maxQueueSize: 1024
      19.              uplink4G:
      20.                numQueueGroups: 0
      21.                numAqsPerGroups: 16
      22.                aqDepthLog2: 4
      23.              downlink4G:
      24.               numQueueGroups: 0
      25.               numAqsPerGroups: 16
      26.               aqDepthLog2: 4
      27.              uplink5G:
      28.               numQueueGroups: 4
      29.               numAqsPerGroups: 16
      30.               aqDepthLog2: 4
      31.              downlink5G:
      32.               numQueueGroups: 4
      33.               numAqsPerGroups: 16
      34.               aqDepthLog2: 4
      1Specify a name for the CR object. The only name that can be specified is config.
      2Specify the node name.
      3Specify the PCI address of the card on which the SR-IOV-FEC Operator will be installed.
      4Specify the number of virtual functions to create. For the Intel vRAN Dedicated Accelerator ACC100, create all 16 VFs.

      The card is configured to provide up to 8 queue groups with up to 16 queues per group. The queue groups can be divided between groups allocated to 5G and 4G and Uplink and Downlink. The Intel vRAN Dedicated Accelerator ACC100 can be configured for:

      • 4G or 5G only

      • 4G and 5G at the same time

      Each configured VF has access to all the queues. Each of the queue groups have a distinct priority level. The request for a given queue group is made from the application level that is, the vRAN application leveraging the FEC device.

    2. Apply the CR:

      1. $ oc apply -f sriovfec_acc100cr.yaml

      After applying the CR, the SR-IOV FEC daemon starts configuring the FEC device.

Verification

  1. Check the status:

    1. $ oc get sriovfecclusterconfig config -o yaml

    Example output

    1. status:
    2.     conditions:
    3.     - lastTransitionTime: "2021-03-19T11:46:22Z"
    4.       message: Configured successfully
    5.       observedGeneration: 1
    6.       reason: Succeeded
    7.       status: "True"
    8.       type: Configured
    9.     inventory:
    10.       sriovAccelerators:
    11.       - deviceID: 0d5c
    12.         driver: pci-pf-stub
    13.         maxVirtualFunctions: 16
    14.         pciAddress: 0000:af:00.0
    15.         vendorID: "8086"
    16.         virtualFunctions:
    17.         - deviceID: 0d5d
    18.           driver: vfio-pci
    19.           pciAddress: 0000:b0:00.0
    20.         - deviceID: 0d5d
    21.           driver: vfio-pci
    22.           pciAddress: 0000:b0:00.1
    23.         - deviceID: 0d5d
    24.           driver: vfio-pci
    25.           pciAddress: 0000:b0:00.2
    26.         - deviceID: 0d5d
    27.           driver: vfio-pci
    28.           pciAddress: 0000:b0:00.3
    29.         - deviceID: 0d5d
    30.           driver: vfio-pci
    31.           pciAddress: 0000:b0:00.4

  2. Check the logs:

    1. Determine the pod name of the SR-IOV daemon:

      1. $ oc get po -o wide | grep sriov-fec-daemonset | grep node1

      Example output

      1. sriov-fec-daemonset-kqqs6                      1/1     Running   0          19h

    2. View the logs:

      1. $ oc logs sriov-fec-daemonset-kqqs6

      Example output

      1. {"level":"Level(-2)","ts":1616794345.4786215,"logger":"daemon.drainhelper.cordonAndDrain()","msg":"node drained"}
      2. {"level":"Level(-4)","ts":1616794345.4786265,"logger":"daemon.drainhelper.Run()","msg":"worker function - start"}
      3. {"level":"Level(-4)","ts":1616794345.5762916,"logger":"daemon.NodeConfigurator.applyConfig","msg":"current node status","inventory":{"sriovAccelerat
      4. ors":[{"vendorID":"8086","deviceID":"0b32","pciAddress":"0000:20:00.0","driver":"","maxVirtualFunctions":1,"virtualFunctions":[]},{"vendorID":"8086"
      5. ,"deviceID":"0d5c","pciAddress":"0000:af:00.0","driver":"","maxVirtualFunctions":16,"virtualFunctions":[]}]}}
      6. {"level":"Level(-4)","ts":1616794345.5763638,"logger":"daemon.NodeConfigurator.applyConfig","msg":"configuring PF","requestedConfig":{"pciAddress":"
      7. 0000:af:00.0","pfDriver":"pci-pf-stub","vfDriver":"vfio-pci","vfAmount":2,"bbDevConfig":{"acc100":{"pfMode":false,"numVfBundles":16,"maxQueueSize":1
      8. 024,"uplink4G":{"numQueueGroups":4,"numAqsPerGroups":16,"aqDepthLog2":4},"downlink4G":{"numQueueGroups":4,"numAqsPerGroups":16,"aqDepthLog2":4},"uplink5G":{"numQueueGroups":0,"numAqsPerGroups":16,"aqDepthLog2":4},"downlink5G":{"numQueueGroups":0,"numAqsPerGroups":16,"aqDepthLog2":4}}}}}
      9. {"level":"Level(-4)","ts":1616794345.5774765,"logger":"daemon.NodeConfigurator.loadModule","msg":"executing command","cmd":"/usr/sbin/chroot /host/ modprobe pci-pf-stub"}
      10. {"level":"Level(-4)","ts":1616794345.5842702,"logger":"daemon.NodeConfigurator.loadModule","msg":"commands output","output":""}
      11. {"level":"Level(-4)","ts":1616794345.5843055,"logger":"daemon.NodeConfigurator.loadModule","msg":"executing command","cmd":"/usr/sbin/chroot /host/ modprobe vfio-pci"}
      12. {"level":"Level(-4)","ts":1616794345.6090655,"logger":"daemon.NodeConfigurator.loadModule","msg":"commands output","output":""}
      13. {"level":"Level(-2)","ts":1616794345.6091156,"logger":"daemon.NodeConfigurator","msg":"device's driver_override path","path":"/sys/bus/pci/devices/0000:af:00.0/driver_override"}
      14. {"level":"Level(-2)","ts":1616794345.6091807,"logger":"daemon.NodeConfigurator","msg":"driver bind path","path":"/sys/bus/pci/drivers/pci-pf-stub/bind"}
      15. {"level":"Level(-2)","ts":1616794345.7488534,"logger":"daemon.NodeConfigurator","msg":"device's driver_override path","path":"/sys/bus/pci/devices/0000:b0:00.0/driver_override"}
      16. {"level":"Level(-2)","ts":1616794345.748938,"logger":"daemon.NodeConfigurator","msg":"driver bind path","path":"/sys/bus/pci/drivers/vfio-pci/bind"}
      17. {"level":"Level(-2)","ts":1616794345.7492096,"logger":"daemon.NodeConfigurator","msg":"device's driver_override path","path":"/sys/bus/pci/devices/0000:b0:00.1/driver_override"}
      18. {"level":"Level(-2)","ts":1616794345.7492566,"logger":"daemon.NodeConfigurator","msg":"driver bind path","path":"/sys/bus/pci/drivers/vfio-pci/bind"}
      19. {"level":"Level(-4)","ts":1616794345.74968,"logger":"daemon.NodeConfigurator.applyConfig","msg":"executing command","cmd":"/sriov_workdir/pf_bb_config ACC100 -c /sriov_artifacts/0000:af:00.0.ini -p 0000:af:00.0"}
      20. {"level":"Level(-4)","ts":1616794346.5203931,"logger":"daemon.NodeConfigurator.applyConfig","msg":"commands output","output":"Queue Groups: 0 5GUL, 0 5GDL, 4 4GUL, 4 4GDL\nNumber of 5GUL engines 8\nConfiguration in VF mode\nPF ACC100 configuration complete\nACC100 PF [0000:af:00.0] configuration complete!\n\n"}
      21. {"level":"Level(-4)","ts":1616794346.520459,"logger":"daemon.NodeConfigurator.enableMasterBus","msg":"executing command","cmd":"/usr/sbin/chroot /host/ setpci -v -s 0000:af:00.0 COMMAND"}
      22. {"level":"Level(-4)","ts":1616794346.5458736,"logger":"daemon.NodeConfigurator.enableMasterBus","msg":"commands output","output":"0000:af:00.0 @04 = 0142\n"}
      23. {"level":"Level(-4)","ts":1616794346.5459251,"logger":"daemon.NodeConfigurator.enableMasterBus","msg":"executing command","cmd":"/usr/sbin/chroot /host/ setpci -v -s 0000:af:00.0 COMMAND=0146"}
      24. {"level":"Level(-4)","ts":1616794346.5795262,"logger":"daemon.NodeConfigurator.enableMasterBus","msg":"commands output","output":"0000:af:00.0 @04 0146\n"}
      25. {"level":"Level(-2)","ts":1616794346.5795407,"logger":"daemon.NodeConfigurator.enableMasterBus","msg":"MasterBus set","pci":"0000:af:00.0","output":"0000:af:00.0 @04 0146\n"}
      26. {"level":"Level(-4)","ts":1616794346.6867144,"logger":"daemon.drainhelper.Run()","msg":"worker function - end","performUncordon":true}
      27. {"level":"Level(-4)","ts":1616794346.6867719,"logger":"daemon.drainhelper.Run()","msg":"uncordoning node"}
      28. {"level":"Level(-4)","ts":1616794346.6896322,"logger":"daemon.drainhelper.uncordon()","msg":"starting uncordon attempts"}
      29. {"level":"Level(-2)","ts":1616794346.69735,"logger":"daemon.drainhelper.uncordon()","msg":"node uncordoned"}
      30. {"level":"Level(-4)","ts":1616794346.6973662,"logger":"daemon.drainhelper.Run()","msg":"cancelling the context to finish the leadership"}
      31. {"level":"Level(-4)","ts":1616794346.7029872,"logger":"daemon.drainhelper.Run()","msg":"stopped leading"}
      32. {"level":"Level(-4)","ts":1616794346.7030034,"logger":"daemon.drainhelper","msg":"releasing the lock (bug mitigation)"}
      33. {"level":"Level(-4)","ts":1616794346.8040674,"logger":"daemon.updateInventory","msg":"obtained inventory","inv":{"sriovAccelerators":[{"vendorID":"8086","deviceID":"0b32","pciAddress":"0000:20:00.0","driver":"","maxVirtualFunctions":1,"virtualFunctions":[]},{"vendorID":"8086","deviceID":"0d5c","pciAddress":"0000:af:00.0","driver":"pci-pf-stub","maxVirtualFunctions":16,"virtualFunctions":[{"pciAddress":"0000:b0:00.0","driver":"vfio-pci","deviceID":"0d5d"},{"pciAddress":"0000:b0:00.1","driver":"vfio-pci","deviceID":"0d5d"}]}]}}
      34. {"level":"Level(-4)","ts":1616794346.9058325,"logger":"daemon","msg":"Update ignored, generation unchanged"}
      35. {"level":"Level(-2)","ts":1616794346.9065044,"logger":"daemon.Reconcile","msg":"Reconciled","namespace":"vran-acceleration-operators","name":"pg-itengdvs02r.altera.com"}

  3. Check the FEC configuration of the card:

    1. $ oc get sriovfecnodeconfig node1 -o yaml

    Example output

    1. status:
    2.     conditions:
    3.     - lastTransitionTime: "2021-03-19T11:46:22Z"
    4.       message: Configured successfully
    5.       observedGeneration: 1
    6.       reason: Succeeded
    7.       status: "True"
    8.       type: Configured
    9.     inventory:
    10.       sriovAccelerators:
    11.       - deviceID: 0d5c (1)
    12.         driver: pci-pf-stub
    13.         maxVirtualFunctions: 16
    14.         pciAddress: 0000:af:00.0
    15.         vendorID: "8086"
    16.         virtualFunctions:
    17.         - deviceID: 0d5d (2)
    18.           driver: vfio-pci
    19.           pciAddress: 0000:b0:00.0
    20.         - deviceID: 0d5d
    21.           driver: vfio-pci
    22.           pciAddress: 0000:b0:00.1
    23.         - deviceID: 0d5d
    24.           driver: vfio-pci
    25.           pciAddress: 0000:b0:00.2
    26.         - deviceID: 0d5d
    27.           driver: vfio-pci
    28.           pciAddress: 0000:b0:00.3
    29.         - deviceID: 0d5d
    30.           driver: vfio-pci
    31.           pciAddress: 0000:b0:00.4
    1The value 0d5c is the deviceID physical function of the FEC device.
    2The value 0d5d is the deviceID virtual function of the FEC device.

Verifying application pod access and FPGA usage on OpenNESS

OpenNESS is an edge computing software toolkit that you can use to onboard and manage applications and network functions on any type of network.

To verify all OpenNESS features are working together, including SR-IOV binding, the device plugin, Wireless Base Band Device (bbdev) configuration, and SR-IOV (FEC) VF functionality inside a non-root pod, you can build an image and run a simple validation application for the device.

For more information, go to openess.org.

Prerequisites

  • Optional: Intel FPGA PAC N3000 card

  • Node or nodes installed with the n3000-operator

  • Node or nodes installed with the SR-IOV-FEC operator

  • Real-Time kernel and huge pages configured with Performance Addon Operator

  • Log in as a user with cluster-admin privileges

Procedure

  1. Create a namespace for the test by completing the following actions:

    1. Define the test-bbdev namespace by creating a file named test-bbdev-namespace.yaml file as shown in the following example:

      1. apiVersion: v1
      2. kind: Namespace
      3. metadata:
      4.   name: test-bbdev
      5.   labels:
      6.     openshift.io/run-level: "1"

    2. Create the namespace by running the following command:

      1. $ oc create -f test-bbdev-namespace.yaml

  2. Create the following Pod specification, and then save the YAML in the pod-test.yaml file:

    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4.   name: pod-bbdev-sample-app
    5.   namespace: test-bbdev (1)
    6. spec:
    7.   containers:
    8.   - securityContext:
    9.       privileged: false
    10.       capabilities:
    11.         add:
    12.          - IPC_LOCK
    13.          - SYS_NICE
    14.     name: bbdev-sample-app
    15.     image: bbdev-sample-app:1.0  (2)
    16.     command: [ "sudo", "/bin/bash", "-c", "--" ]
    17.     runAsUser: 0 (3)
    18.     resources:
    19.       requests:
    20.         hugepages-1Gi: 4Gi (4)
    21.         memory: 1Gi
    22.         cpu: "4" (5)
    23.         intel.com/intel_fec_5g: '1' (6)
    24.         #intel.com/intel_fec_acc100: '1'
    25.         #intel.com/intel_fec_lte: '1'
    26.       limits:
    27.         memory: 4Gi
    28.         cpu: "4"
    29.         hugepages-1Gi: 4Gi
    30.         intel.com/intel_fec_5g: '1'
    31.         #intel.com/intel_fec_acc100: '1'
    32.         #intel.com/intel_fec_lte: '1
    1Specify the namespace you created in step 1.
    2This defines the test image containing the compiled DPDK.
    3Make the container execute internally as the root user.
    4Specify hugepage size hugepages-1Gi and the quantity of hugepages that will be allocated to the pod. Hugepages and isolated CPUs need to be configured using the Performance Addon Operator.
    5Specify the number of CPUs.
    6Testing of the N3000 5G FEC configuration is supported by intel.com/intel_fec_5g.

    To test the ACC100 configuration, uncomment intel.com/intel_fec_acc100 by removing the # symbol. To test the N3000 4G/LTE configuration, uncomment intel.com/intel_fec_lte by removing the # symbol. Only one resource can be active at any time.

  3. Create the pod:

    1. $ oc apply -f pod-test.yaml

  4. Check that the pod is created:

    1. $ oc get pods -n test-bbdev

    Example output

    1. NAME                                            READY           STATUS          RESTARTS        AGE
    2. pod-bbdev-sample-app                            1/1             Running         0               80s

  5. Use a remote shell to log in to the pod-bbdev-sample-app:

    1. $ oc rsh pod-bbdev-sample-app

    Example output

    1. sh-4.4#

  6. Print a list of environment variables:

    1. sh-4.4# env

    Example output

    1. N3000_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP_ADDR=172.30.133.131
    2. SRIOV_FEC_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP_PROTO=tcp
    3. DPDK_VERSION=20.11
    4. PCIDEVICE_INTEL_COM_INTEL_FEC_5G=0.0.0.0:1d.00.0 (1)
    5. ~/usr/bin/env
    6. HOSTNAME=fec-pod
    1This is the PCI address of the virtual function. Depending on the resource that you requested in the pod-test.yaml file, this can be any one of following three PCI addresses:

    • PCIDEVICE_INTEL_COM_INTEL_FEC_ACC100

    • PCIDEVICE_INTEL_COM_INTEL_FEC_5G

    • PCIDEVICE_INTEL_COM_INTEL_FEC_LTE

  7. Change to the test-bbdev directory:

    1. sh-4.4# cd test/test-bbdev/

    The directory is in the pod and not on your local computer.

  8. Check the CPUs that are assigned to the pod:

    1. sh-4.4# export CPU=$(cat /sys/fs/cgroup/cpuset/cpuset.cpus)
    2. sh-4.4# echo ${CPU}

    This prints out the CPUs that are assigned to the fec.pod.

    Example output

    1. 24,25,64,65

  9. Run the test-bbdev application to test the device:

    1. sh-4.4# ./test-bbdev.py -e="-l ${CPU} -a ${PCIDEVICE_INTEL_COM_INTEL_FEC_5G}" -c validation \ -n 64 -b 32 -l 1 -v ./test_vectors/*"

    Example output

    ``` Executing: ../../build/app/dpdk-test-bbdev -l 24-25,64-65 0000:1d.00.0 — -n 64 -l 1 -c validation -v ./test_vectors/bbdev_null.data -b 32 EAL: Detected 80 lcore(s) EAL: Detected 2 NUMA nodes Option -w, —pci-whitelist is deprecated, use -a, —allow option instead EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode ‘VA’ EAL: Probing VFIO support… EAL: VFIO support initialized EAL: using IOMMU type 1 (Type 1) EAL: Probe PCI driver: intel_fpga_5ngr_fec_vf (8086:d90) device: 0000:1d.00.0 (socket 1) EAL: No legacy callbacks, legacy socket not created

  1. ===========================================================
  2. Starting Test Suite : BBdev Validation Tests
  3. Test vector file = ldpc_dec_v7813.data
  4. Device 0 queue 16 setup failed
  5. Allocated all queues (id=16) at prio0 on dev0
  6. Device 0 queue 32 setup failed
  7. Allocated all queues (id=32) at prio1 on dev0
  8. Device 0 queue 48 setup failed
  9. Allocated all queues (id=48) at prio2 on dev0
  10. Device 0 queue 64 setup failed
  11. Allocated all queues (id=64) at prio3 on dev0
  12. Device 0 queue 64 setup failed
  13. All queues on dev 0 allocated: 64
  14. + ------------------------------------------------------- +
  15. == test: validation
  16. dev:0000:b0:00.0, burst size: 1, num ops: 1, op type: RTE_BBDEV_OP_LDPC_DEC
  17. Operation latency:
  18.         avg: 23092 cycles, 10.0838 us
  19.         min: 23092 cycles, 10.0838 us
  20.         max: 23092 cycles, 10.0838 us
  21. TestCase [ 0] : validation_tc passed
  22.  + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +
  23.  + Test Suite Summary : BBdev Validation Tests
  24.  + Tests Total :        1
  25.  + Tests Skipped :      0
  26.  + Tests Passed :       1 (1)
  27.  + Tests Failed :       0
  28.  + Tests Lasted :       177.67 ms
  29.  + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +
  30. ```
  32. <table><tbody><tr><td><i data-value="1"></i><b>1</b></td><td>While some tests can be skipped, be sure that the vector tests pass.</td></tr></tbody></table>

Additional resources