Distributed Tracing

Overview of OpenTelemetry

OpenTelemetry is a collection of tools, APIs, and SDKs. You can use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) for analysis in order to understand your software’s performance and behavior.

Requirements

  • set tracer
  • different exporter
  • start and end span in server

Design Details

  • SpanProcessor: BatchSpanProcessor

  • Exporter: log(default), would be changed from properties

  1. // Configure the batch spans processor. This span processor exports span in batches.
  2. BatchSpanProcessor batchSpansProcessor =
  3.     BatchSpanProcessor.builder(exporter)
  4.         .setMaxExportBatchSize(512) // set the maximum batch size to use
  5.         .setMaxQueueSize(2048) // set the queue size. This must be >= the export batch size
  6.         .setExporterTimeout(
  7.             30, TimeUnit.SECONDS) // set the max amount of time an export can run before getting
  8.         // interrupted
  9.         .setScheduleDelay(5, TimeUnit.SECONDS) // set time between two different exports
  10.         .build();
  11. OpenTelemetrySdk.builder()
  12.     .setTracerProvider(
  13.         SdkTracerProvider.builder().addSpanProcessor(batchSpansProcessor).build())
  14.     .build();
  1. When using the method ‘init()’ of the class “EventMeshHTTPServer”, the class “AbstractHTTPServer” will get the tracer
  1. super.openTelemetryTraceFactory = new OpenTelemetryTraceFactory(eventMeshHttpConfiguration);
  2. super.tracer = openTelemetryTraceFactory.getTracer(this.getClass().toString());
  3. super.textMapPropagator = openTelemetryTraceFactory.getTextMapPropagator();
  1. then the trace in class “AbstractHTTPServer” will work.

Problems

How to set different exporter in class ‘OpenTelemetryTraceFactory’? (Solved)

After I get the exporter type from properties, how to deal with it.

The ‘logExporter’ only needs to new it.

But the ‘zipkinExporter’ needs to new and use the “getZipkinExporter()” method.

Solutions

Solution of different exporter

Use reflection to get an exporter.

First of all, different exporter must implement the interface ‘EventMeshExporter’.

Then we get the exporter name from the configuration and reflect to the class.

  1. //different spanExporter
  2. String exporterName = configuration.eventMeshTraceExporterType;
  3. //use reflection to get spanExporter
  4. String className = String.format("org.apache.eventmesh.runtime.exporter.%sExporter",exporterName);
  5. EventMeshExporter eventMeshExporter = (EventMeshExporter) Class.forName(className).newInstance();
  6. spanExporter = eventMeshExporter.getSpanExporter(configuration);

Additional, this will surround with try catch.If the specified exporter cannot be obtained successfully, the default exporter log will be used instead

Improvement of different exporter

SPI (To be completed)

Appendix

References

https://github.com/open-telemetry/docs-cn/blob/main/QUICKSTART.md

https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/netty