VCL Socket Interface

Attention

The VCL socket interface extension is experimental and is currently under active development.

This socket interface extension provides Envoy with high speed L2-L7 user space networking by integrating with fd.io VPP through VPP’s Comms Library (VCL).

The VCL socket interface is only included in contrib images

Example configuration

  1. bootstrap_extensions:
  2.   - name: envoy.extensions.vcl.vcl_socket_interface
  3.     typed_config:
  4.       "@type": type.googleapis.com/envoy.extensions.vcl.v3alpha.VclSocketInterface
  5. default_socket_interface: "envoy.extensions.vcl.vcl_socket_interface"

How it works

If enabled, the extension attaches through a VCL interface (vcl_interface.h) to VCL, and consequently to the external VPP process, when it is initialized during Envoy bootstrap. This registers a main VCL worker, while subsequent Envoy workers are registered whenever the socket interface extension detects that its code is being executed by a pthread that has not yet been registered with VCL.

Because both libevent and VCL want to handle the async polling and the dispatching of IoHandles, the VCL interface delegates control to libevent by registering with it, for each Envoy worker, the eventfd associated to the VCL worker’s VPP message queue. These shared memory message queues are used by VPP to convey io/ctrl events to VCL and the eventfds are used to signal message queue transitions from empty to non-empty state. This ultimately means that VPP generated events force libevent to hand over control to the VCL interface which, for each Envoy worker, uses an internally maintained epoll fd to poll/pull events from VCL and subsequently dispatch them. To support all of these indirect interactions, the socket interface makes use of custom IoHandle and FileEvent implementations that convert between Envoy and VCL API calls.

Installing and running VPP/VCL

For information on how to build and/or install VPP see the getting started guide here. Assuming the use of DPDK interfaces, a minimal startup.conf file that also configures the host stack would consist of:

  1. unix {
  2.   # Run in interactive mode and not as a daemon
  3.   nodaemon
  4.   interactive
  6.   # Cli socket to be used by vppctl
  7.   cli-listen /run/vpp/cli.sock
  9.   # Group id is an example
  10.   gid vpp
  11. }
  13. cpu {
  14.   # Avoid using core 0 and run vpp's main thread on core 1
  15.   skip-cores 0
  16.   main-core 1
  18.   # Set logical CPU core(s) where worker threads are running. For performance testing make
  19.   # sure the cores are on the same numa as the NIC(s). Use lscpu to determine the numa of
  20.   # a cpu and "sh hardware" in vpp cli to determine the numa of a NIC. To configure multiple
  21.   # workers lists are also possible, e.g., corelist-workers 2-4,6
  22.   corelist-workers 2
  23. }
  25. buffers {
  26.   # Default is 16384 (8192 if running unpriviledged)
  27.   buffers-per-numa 16384
  28. }
  30. dpdk {
  31.   # Notes:
  32.   # - Assuming only one NIC is used
  33.   # - The PCI address is an example, the actual one should be found using something like dpdk_devbind
  34.   #    https://github.com/DPDK/dpdk/blob/main/usertools/dpdk-devbind.py
  35.   # - Number of rx queues (num-rx-queus) should be number of workers
  36.   dev 0000:18:00.0 {
  37.     num-tx-desc 256
  38.     num-rx-desc 256
  39.     num-rx-queues 1
  40.   }
  41. }
  43. session {
  44.   # Use session layer socket api for VCL attachments
  45.   use-app-socket-api
  47.   # VPP worker's message queues lengths
  48.   event-queue-length 100000
  49. }

Manually start VPP, once a binary is obtained: ./vpp -c startup.conf

VCL can be configured by either adding a configuration file to /etc/vpp/vcl.conf or by pointing the VCL_CONFIG environment variable to a configuration file. A minimal example that can be used for RPS load testing can be found lower:

  1. vcl {
  2.   # Max rx/tx session buffers sizes in bytes. Increase for high throughput traffic.
  3.   rx-fifo-size 400000
  4.   tx-fifo-size 400000
  6.   # Size of shared memory segments between VPP and VCL in bytes
  7.   segment-size 1000000000
  9.   # App has access to global routing table
  10.   app-scope-global
  12.   # Allow inter-app shared-memory cut-through sessions
  13.   app-scope-local
  15.   # Pointer to session layer's socket api socket
  16.   app-socket-api /var/run/vpp/app_ns_sockets/default
  18.   # Message queues use eventfds for notifications
  19.   use-mq-eventfd
  21.   # VCL worker incoming message queue size
  22.   event-queue-size 40000
  23. }