Traffic tapping

Envoy currently provides two experimental extensions that can tap traffic:

Tap transport socket configuration

Attention

The tap transport socket is experimental and is currently under active development. There is currently a very limited set of match conditions, output configuration, output sinks, etc. Capabilities will be expanded over time and the configuration structures are likely to change.

Tapping can be configured on Listener and Cluster transport sockets, providing the ability to interpose on downstream and upstream L4 connections respectively.

To configure traffic tapping, add an envoy.transport_sockets.tap transport socket configuration to the listener or cluster. For a plain text socket this might look like:

  1. transport_socket:
  2.   name: envoy.transport_sockets.tap
  3.   typed_config:
  4.     "@type": type.googleapis.com/envoy.extensions.transport_sockets.tap.v3.Tap
  5.     common_config:
  6.       static_config:
  7.         match_config:
  8.           any_match: true
  9.         output_config:
  10.           sinks:
  11.             - format: PROTO_BINARY
  12.               file_per_tap:
  13.                 path_prefix: /some/tap/path
  14.     transport_socket:
  15.       name: envoy.transport_sockets.raw_buffer
  16.       typed_config:
  17.         "@type": type.googleapis.com/envoy.extensions.transport_sockets.raw_buffer.v3.RawBuffer

For a TLS socket, this will be:

  1. transport_socket:
  2.   name: envoy.transport_sockets.tap
  3.   typed_config:
  4.     "@type": type.googleapis.com/envoy.extensions.transport_sockets.tap.v3.Tap
  5.     common_config:
  6.       static_config:
  7.         match_config:
  8.           any_match: true
  9.         output_config:
  10.           sinks:
  11.             - format: PROTO_BINARY
  12.               file_per_tap:
  13.                 path_prefix: /some/tap/path
  14.     transport_socket:
  15.       name: envoy.transport_sockets.tls
  16.       typed_config: <TLS context>

where the TLS context configuration replaces any existing downstream or upstream TLS configuration on the listener or cluster, respectively.

Each unique socket instance will generate a trace file prefixed with path_prefix. E.g. /some/tap/path_0.pb.

Buffered data limits

For buffered socket taps, Envoy will limit the amount of body data that is tapped to avoid OOM situations. The default limit is 1KiB for both received and transmitted data. This is configurable via the max_buffered_rx_bytes and max_buffered_tx_bytes settings. When a buffered socket tap is truncated, the trace will indicate truncation via the read_truncated and write_truncated fields as well as the body truncated field.

Streaming

The tap transport socket supports both buffered and streaming, controlled by the streaming setting. When buffering, SocketBufferedTrace messages are emitted. When streaming, a series of SocketStreamedTraceSegment are emitted.

See the HTTP tap filter streaming documentation for more information. Most of the concepts overlap between the HTTP filter and the transport socket.

PCAP generation

The generated trace file can be converted to libpcap format, suitable for analysis with tools such as Wireshark with the tap2pcap utility, e.g.:

  1. bazel run @envoy_api//tools:tap2pcap /some/tap/path_0.pb path_0.pcap
  2. tshark -r path_0.pcap -d "tcp.port==10000,http2" -P
  3.   1   0.000000    127.0.0.1 → 127.0.0.1    HTTP2 157 Magic, SETTINGS, WINDOW_UPDATE, HEADERS
  4.   2   0.013713    127.0.0.1 → 127.0.0.1    HTTP2 91 SETTINGS, SETTINGS, WINDOW_UPDATE
  5.   3   0.013820    127.0.0.1 → 127.0.0.1    HTTP2 63 SETTINGS
  6.   4   0.128649    127.0.0.1 → 127.0.0.1    HTTP2 5586 HEADERS
  7.   5   0.130006    127.0.0.1 → 127.0.0.1    HTTP2 7573 DATA
  8.   6   0.131044    127.0.0.1 → 127.0.0.1    HTTP2 3152 DATA, DATA