trace peer forwarder

The trace_peer_forwarder processor is used with peer forwarder to reduce by half the number of events forwarded in a Trace Analytics pipeline. In Trace Analytics, each event is typically duplicated when it is sent from otel-trace-pipeline to raw-pipeline and service-map-pipeline. When pipelines forward events, this causes the core peer forwarder to send multiple HTTP requests for the same event. You can use trace peer forwarder to forward an event once through the otel-trace-pipeline instead of raw-pipeline and service-map-pipeline, which prevents unnecessary HTTP requests.

You should use trace_peer_forwarder for Trace Analytics pipelines when you have multiple nodes.

Usage

To get started with trace_peer_forwarder, first configure peer forwarder. Then create a pipeline.yaml file and specify trace peer forwarder as the processor. You can configure peer forwarder in your data-prepper-config.yaml file. For more detailed information, see Configuring Data Prepper.

See the following example pipeline.yaml file:

  1. otel-trace-pipeline:
  2.   delay: "100"
  3.   source:
  4.     otel_trace_source:
  5.   processor:
  6.     - trace_peer_forwarder:
  7.   sink:
  8.     - pipeline:
  9.         name: "raw-pipeline"
  10.     - pipeline:
  11.         name: "service-map-pipeline"
  12. raw-pipeline:
  13.   source:
  14.     pipeline:
  15.       name: "entry-pipeline"
  16.   processor:
  17.     - otel_trace_raw:
  18.   sink:
  19.     - opensearch:
  20. service-map-pipeline:
  21.   delay: "100"
  22.   source:
  23.     pipeline:
  24.       name: "entry-pipeline"
  25.   processor:
  26.     - service_map_stateful:
  27.   sink:
  28.     - opensearch:

In the preceding pipeline.yaml file, events are forwarded in the otel-trace-pipeline to the target peer, and no forwarding is performed in raw-pipeline or service-map-pipeline. This process helps improve network performance by forwarding events (as HTTP requests) once instead of twice.