XRay Instrumentation

Introduction

XRay is a function call tracing system which combines compiler-insertedinstrumentation points and a runtime library that can dynamically enable anddisable the instrumentation.

More high level information about XRay can be found in the XRay whitepaper.

This document describes how to use XRay as implemented in LLVM.

XRay in LLVM

XRay consists of three main parts:

• Compiler-inserted instrumentation points.

• A runtime library for enabling/disabling tracing at runtime.

• A suite of tools for analysing the traces.

NOTE: As of July 25, 2018 , XRay is only available for the followingarchitectures running Linux: x86_64, arm7 (no thumb), aarch64, powerpc64le,mips, mipsel, mips64, mips64el, NetBSD: x86_64, FreeBSD: x86_64 andOpenBSD: x86_64.

The compiler-inserted instrumentation points come in the form of nop-sleds inthe final generated binary, and an ELF section named xray_instr_map whichcontains entries pointing to these instrumentation points. The runtime libraryrelies on being able to access the entries of the xray_instr_map, andoverwrite the instrumentation points at runtime.

Using XRay

You can use XRay in a couple of ways:

• Instrumenting your C/C++/Objective-C/Objective-C++ application.
• Generating LLVM IR with the correct function attributes.

The rest of this section covers these main ways and later on how to customisewhat XRay does in an XRay-instrumented binary.

Instrumenting your C/C++/Objective-C Application

The easiest way of getting XRay instrumentation for your application is byenabling the -fxray-instrument flag in your clang invocation.

For example:

clang -fxray-instrument ...

By default, functions that have at least 200 instructions will get XRayinstrumentation points. You can tweak that number through the-fxray-instruction-threshold= flag:

clang -fxray-instrument -fxray-instruction-threshold=1 ...

You can also specifically instrument functions in your binary to either alwaysor never be instrumented using source-level attributes. You can do it using theGCC-style attributes or C++11-style attributes.

[[clang::xray_always_instrument]] void always_instrumented();
 
[[clang::xray_never_instrument]] void never_instrumented();
 
void alt_always_instrumented() __attribute__((xray_always_instrument));
 
void alt_never_instrumented() __attribute__((xray_never_instrument));

When linking a binary, you can either manually link in the XRay RuntimeLibrary or use clang to link it in automatically with the-fxray-instrument flag. Alternatively, you can statically link-in the XRayruntime library from compiler-rt – those archive files will take the name oflibclang_rt.xray-{arch} where {arch} is the mnemonic supported by clang(x86_64, arm7, etc.).

LLVM Function Attribute

If you’re using LLVM IR directly, you can add the function-instrumentstring attribute to your functions, to get the similar effect that theC/C++/Objective-C source-level attributes would get:

define i32 @always_instrument() uwtable "function-instrument"="xray-always" {
  ; ...
}
 
define i32 @never_instrument() uwtable "function-instrument"="xray-never" {
  ; ...
}

You can also set the xray-instruction-threshold attribute and provide anumeric string value for how many instructions should be in the function beforeit gets instrumented.

define i32 @maybe_instrument() uwtable "xray-instruction-threshold"="2" {
  ; ...
}

Special Case File

Attributes can be imbued through the use of special case files instead ofadding them to the original source files. You can use this to mark certainfunctions and classes to be never, always, or instrumented with first-argumentlogging from a file. The file’s format is described below:

# Comments are supported
[always]
fun:always_instrument
fun:log_arg1=arg1 # Log the first argument for the function
 
[never]
fun:never_instrument

These files can be provided through the -fxray-attr-list= flag to clang.You may have multiple files loaded through multiple instances of the flag.

XRay Runtime Library

The XRay Runtime Library is part of the compiler-rt project, which implementsthe runtime components that perform the patching and unpatching of insertedinstrumentation points. When you use clang to link your binaries and the-fxray-instrument flag, it will automatically link in the XRay runtime.

The default implementation of the XRay runtime will enable XRay instrumentationbefore main starts, which works for applications that have a shortlifetime. This implementation also records all function entry and exit eventswhich may result in a lot of records in the resulting trace.

Also by default the filename of the XRay trace is xray-log.XXXXXX where theXXXXXX part is randomly generated.

These options can be controlled through the XRAY_OPTIONS environmentvariable, where we list down the options and their defaults below.

If you choose to not use the default logging implementation that comes with theXRay runtime and/or control when/how the XRay instrumentation runs, you may usethe XRay APIs directly for doing so. To do this, you’ll need to include thexray_log_interface.h from the compiler-rt xray directory. The important APIfunctions we list below:

• __xray_log_register_mode(…): Register a logging implementation againsta string Mode identifier. The implementation is an instance ofXRayLogImpl defined in xray/xray_log_interface.h.
• xray_log_select_mode(…): Select the mode to install, associated witha string Mode identifier. Only implementations registered withxray_log_register_mode(…) can be chosen with this function.
• __xray_log_init_mode(…): This function allows for initializing andre-initializing an installed logging implementation. Seexray/xray_log_interface.h for details, part of the XRay compiler-rtinstallation.

Once a logging implementation has been initialized, it can be “stopped” byfinalizing the implementation through the xray_log_finalize() function.The finalization routine is the opposite of the initialization. When finalized,an implementation’s data can be cleared out through thexray_log_flushLog() function. For implementations that support in-memoryprocessing, these should register an iterator function to provide access to thedata via the xray_log_set_buffer_iterator(…) which allows code callingthe xray_log_process_buffers(…) function to deal with the data inmemory.

All of this is better explained in the xray/xray_log_interface.h header.

Basic Mode

XRay supports a basic logging mode which will trace the application’sexecution, and periodically append to a single log. This mode can beinstalled/enabled by setting xray_mode=xray-basic in the XRAY_OPTIONSenvironment variable. Combined with patch_premain=true this can allow fortracing applications from start to end.

Like all the other modes installed through xray_log_select_mode(…), theimplementation can be configured through the xray_log_init_mode(…)function, providing the mode string and the flag options. Basic-mode specificdefaults can be provided in the XRAY_BASIC_OPTIONS environment variable.

Flight Data Recorder Mode

XRay supports a logging mode which allows the application to only capture afixed amount of memory’s worth of events. Flight Data Recorder (FDR) mode worksvery much like a plane’s “black box” which keeps recording data to memory in afixed-size circular queue of buffers, and have the data availableprogrammatically until the buffers are finalized and flushed. To use FDR modeon your application, you may set the xray_mode variable to xray-fdr inthe XRAY_OPTIONS environment variable. Additional options to the FDR modeimplementation can be provided in the XRAY_FDR_OPTIONS environmentvariable. Programmatic configuration can be done by calling__xray_log_init_mode("xray-fdr", <configuration string>) once it has beenselected/installed.

When the buffers are flushed to disk, the result is a binary trace formatdescribed by XRay FDR format

When FDR mode is on, it will keep writing and recycling memory buffers untilthe logging implementation is finalized – at which point it can be flushed andre-initialised later. To do this programmatically, we follow the workflowprovided below:

// Patch the sleds, if we haven't yet.
auto patch_status = __xray_patch();
 
// Maybe handle the patch_status errors.
 
// When we want to flush the log, we need to finalize it first, to give
// threads a chance to return buffers to the queue.
auto finalize_status = __xray_log_finalize();
if (finalize_status != XRAY_LOG_FINALIZED) {
  // maybe retry, or bail out.
}
 
// At this point, we are sure that the log is finalized, so we may try
// flushing the log.
auto flush_status = __xray_log_flushLog();
if (flush_status != XRAY_LOG_FLUSHED) {
  // maybe retry, or bail out.
}

The default settings for the FDR mode implementation will create logs namedsimilarly to the basic log implementation, but will have a different logformat. All the trace analysis tools (and the trace reading library) willsupport all versions of the FDR mode format as we add more functionality andrecord types in the future.

NOTE: We do not promise perpetual support for when we update the logversions we support going forward. Deprecation of the formats will beannounced and discussed on the developers mailing list.

Trace Analysis Tools

We currently have the beginnings of a trace analysis tool in LLVM, which can befound in the tools/llvm-xray directory. The llvm-xray tool currentlysupports the following subcommands:

• extract: Extract the instrumentation map from a binary, and return it asYAML.
• account: Performs basic function call accounting statistics with variousoptions for sorting, and output formats (supports CSV, YAML, andconsole-friendly TEXT).
• convert: Converts an XRay log file from one format to another. We canconvert from binary XRay traces (both basic and FDR mode) to YAML,flame-graph friendly textformats, as well as Chrome Trace Viewer (catapult) formats.
• graph: Generates a DOT graph of the function call relationships betweenfunctions found in an XRay trace.
• stack: Reconstructs function call stacks from a timeline of functioncalls in an XRay trace.These subcommands use various library components found as part of the XRaylibraries, distributed with the LLVM distribution. These are:

• llvm/XRay/Trace.h : A trace reading library for conveniently loadingan XRay trace of supported forms, into a convenient in-memory representation.All the analysis tools that deal with traces use this implementation.
• llvm/XRay/Graph.h : A semi-generic graph type used by the graphsubcommand to conveniently represent a function call graph with statisticsassociated with edges and vertices.
• llvm/XRay/InstrumentationMap.h: A convenient tool for analyzing theinstrumentation map in XRay-instrumented object files and binaries. Theextract and stack subcommands uses this particular library.

Future Work

There are a number of ongoing efforts for expanding the toolset building aroundthe XRay instrumentation system.

Trace Analysis Tools

• Work is in progress to integrate with or develop tools to visualize findingsfrom an XRay trace. Particularly, the stack tool is being expanded tooutput formats that allow graphing and exploring the duration of time in eachcall stack.
• With a large instrumented binary, the size of generated XRay traces canquickly become unwieldy. We are working on integrating pruning techniques andheuristics for the analysis tools to sift through the traces and surface onlyrelevant information.

More Platforms

We’re looking forward to contributions to port XRay to more architectures andoperating systems.