How to Cross Compile Compiler-rt Builtins For Arm

Introduction

This document contains information about building and testing the builtins partof compiler-rt for an Arm target, from an x86_64 Linux machine.

While this document concentrates on Arm and Linux the general principles shouldapply to other targets supported by compiler-rt. Further contributions for othertargets are welcome.

The instructions in this document depend on libraries and programs external toLLVM, there are many ways to install and configure these dependencies so youmay need to adapt the instructions here to fit your own local situation.

Prerequisites

In this use case we’ll be using cmake on a Debian-based Linux system,cross-compiling from an x86_64 host to a hard-float Armv7-A target. We’ll beusing as many of the LLVM tools as we can, but it is possible to use GNUequivalents.

  • A build of LLVM/clang for the llvm-tools and llvm-config
  • A clang executable with support for the ARM target
  • compiler-rt sources
  • The qemu-arm user mode emulator
  • An arm-linux-gnueabihf sysroot

In this example we will be using ninja.

See https://compiler-rt.llvm.org/ for more information about the dependencieson clang and LLVM.

See https://llvm.org/docs/GettingStarted.html for information about obtainingthe source for LLVM and compiler-rt. Note that the getting started guideplaces compiler-rt in the projects subdirectory, but this is not essential andif you are using the BaremetalARM.cmake cache for v6-M, v7-M and v7-EM thencompiler-rt must be placed in the runtimes directory.

qemu-arm should be available as a package for your Linux distribution.

The most complicated of the prerequisites to satisfy is the arm-linux-gnueabihfsysroot. In theory it is possible to use the Linux distributions multiarchsupport to fulfill the dependencies for building but unfortunately due to/usr/local/include being added some host includes are selected. The easiest wayto supply a sysroot is to download the arm-linux-gnueabihf toolchain. This canbe found at: https://developer.arm.com/open-source/gnu-toolchain/gnu-a/downloads for gcc 8 and above https://releases.linaro.org/components/toolchain/binaries/ for gcc 4.9 to 7.3

Building compiler-rt builtins for Arm

We will be doing a standalone build of compiler-rt using the following cmakeoptions.

  • path/to/compiler-rt
  • -G Ninja
  • -DCOMPILER_RT_BUILD_BUILTINS=ON
  • -DCOMPILER_RT_BUILD_SANITIZERS=OFF
  • -DCOMPILER_RT_BUILD_XRAY=OFF
  • -DCOMPILER_RT_BUILD_LIBFUZZER=OFF
  • -DCOMPILER_RT_BUILD_PROFILE=OFF
  • -DCMAKE_C_COMPILER=/path/to/clang
  • -DCMAKE_AR=/path/to/llvm-ar
  • -DCMAKE_NM=/path/to/llvm-nm
  • -DCMAKE_RANLIB=/path/to/llvm-ranlib
  • -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld"
  • -DCMAKE_C_COMPILER_TARGET="arm-linux-gnueabihf"
  • -DCMAKE_ASM_COMPILER_TARGET="arm-linux-gnueabihf"
  • -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON
  • -DLLVM_CONFIG_PATH=/path/to/llvm-config
  • -DCMAKE_C_FLAGS="build-c-flags"
  • -DCMAKE_ASM_FLAGS="build-c-flags"

The build-c-flags need to be sufficient to pass the C-make compiler check,compile compiler-rt, and if you are running the tests, compile and link thetests. When cross-compiling with clang we will need to pass sufficientinformation to generate code for the Arm architecture we are targeting. We willneed to select the Arm target, select the Armv7-A architecture and choosebetween using Arm or Thumb.instructions. For example:

  • —target=arm-linux-gnueabihf
  • -march=armv7a
  • -mthumb

When using a GCC arm-linux-gnueabihf toolchain the following flags areneeded to pick up the includes and libraries:

  • —gcc-toolchain=/path/to/dir/toolchain
  • —sysroot=/path/to/toolchain/arm-linux-gnueabihf/libc

In this example we will be adding all of the command line options to bothCMAKE_C_FLAGS and CMAKE_ASM_FLAGS. There are cmake flags to pass some ofthese options individually which can be used to simplify the build-c-flags:

  • -DCMAKE_C_COMPILER_TARGET="arm-linux-gnueabihf"
  • -DCMAKE_ASM_COMPILER_TARGET="arm-linux-gnueabihf"
  • -DCMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN=/path/to/dir/toolchain
  • -DCMAKE_SYSROOT=/path/to/dir/toolchain/arm-linux-gnueabihf/libc

Once cmake has completed the builtins can be built with ninja builtins

Testing compiler-rt builtins using qemu-arm

To test the builtins library we need to add a few more cmake flags to enabletesting and set up the compiler and flags for test case. We must also tellcmake that we wish to run the tests on qemu-arm.

  • -DCOMPILER_RT_EMULATOR="qemu-arm -L /path/to/armhf/sysroot
  • -DCOMPILER_RT_INCLUDE_TESTS=ON
  • -DCOMPILER_RT_TEST_COMPILER="/path/to/clang"
  • -DCOMPILER_RT_TEST_COMPILER_CFLAGS="test-c-flags"

The /path/to/armhf/sysroot should be the same as the one passed to—sysroot in the “build-c-flags”.

The “test-c-flags” need to include the target, architecture, gcc-toolchain,sysroot and arm/thumb state. The additional cmake defines such asCMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN do not apply when building the tests. Ifyou have put all of these in “build-c-flags” then these can be repeated. If youwish to use lld to link the tests then add "-fuse-ld=lld.

Once cmake has completed the tests can be built and run usingninja check-builtins

Troubleshooting

The cmake try compile stage fails

At an early stage cmake will attempt to compile and link a simple C program totest if the toolchain is working.

This stage can often fail at link time if the —sysroot and—gcc-toolchain options are not passed to the compiler. Check theCMAKE_C_FLAGS and CMAKE_C_COMPILER_TARGET flags.

It can be useful to build a simple example outside of cmake with your toolchainto make sure it is working. For example: clang —target=arm-linux-gnueabi -march=armv7a —gcc-toolchain=/path/to/gcc-toolchain —sysroot=/path/to/gcc-toolchain/arm-linux-gnueabihf/libc helloworld.c

Clang uses the host header files

On debian based systems it is possible to install multiarch support forarm-linux-gnueabi and arm-linux-gnueabihf. In many cases clang can successfullyuse this multiarch support when -gcc-toolchain and –sysroot are not supplied.Unfortunately clang adds /usr/local/include before/usr/include/arm-linux-gnueabihf leading to errors when compiling the hostsheader files.

The multiarch support is not sufficient to build the builtins you will need touse a separate arm-linux-gnueabihf toolchain.

No target passed to clang

If clang is not given a target it will typically use the host target, this willnot understand the Arm assembly language files resulting in error messages suchas error: unknown directive .syntax unified.

You can check the clang invocation in the error message to see if there is no—target or if it is set incorrectly. The cause is usuallyCMAKE_ASM_FLAGS not containing —target or CMAKE_ASM_COMPILER_TARGET not being present.

Arm architecture not given

The —target=arm-linux-gnueabihf will default to arm architecture v4t whichcannot assemble the barrier instructions used in the synch_and_fetch sourcefiles.

The cause is usually a missing -march=armv7a from the CMAKE_ASM_FLAGS.

Compiler-rt builds but the tests fail to build

The flags used to build the tests are not the same as those used to build thebuiltins. The c flags are provided by COMPILER_RT_TEST_COMPILE_CFLAGS andthe CMAKE_C_COMPILER_TARGET, CMAKE_ASM_COMPILER_TARGET,CMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN and CMAKE_SYSROOT flags are notapplied.

Make sure that COMPILER_RT_TEST_COMPILE_CFLAGS contains all the necessaryinformation.

Modifications for other Targets

Arm Soft-Float Target

The instructions for the Arm hard-float target can be used for the soft-floattarget by substituting soft-float equivalents for the sysroot and target. Thetarget to use is:

  • -DCMAKE_C_COMPILER_TARGET=arm-linux-gnueabi

Depending on whether you want to use floating point instructions or not youmay need extra c-flags such as -mfloat-abi=softfp for use of floating-pointinstructions, and -mfloat-abi=soft -mfpu=none for software floating-pointemulation.

You will need to use an arm-linux-gnueabi GNU toolchain for soft-float.

AArch64 Target

The instructions for Arm can be used for AArch64 by substituting AArch64equivalents for the sysroot, emulator and target.

  • -DCMAKE_C_COMPILER_TARGET=aarch64-linux-gnu
  • -DCOMPILER_RT_EMULATOR="qemu-aarch64 -L /path/to/aarch64/sysroot

The CMAKE_C_FLAGS and COMPILER_RT_TEST_COMPILER_CFLAGS may also need:"—sysroot=/path/to/aarch64/sysroot —gcc-toolchain=/path/to/gcc-toolchain"

Armv6-m, Armv7-m and Armv7E-M targets

To build and test the libraries using a similar method to Armv7-A is possiblebut more difficult. The main problems are:

  • There isn’t a qemu-arm user-mode emulator for bare-metal systems. The qemu-system-arm can be used but this is significantly more difficult to setup.
  • The targets to compile compiler-rt have the suffix -none-eabi. This uses the BareMetal driver in clang and by default won’t find the libraries needed to pass the cmake compiler check.

As the Armv6-M, Armv7-M and Armv7E-M builds of compiler-rt only use instructionsthat are supported on Armv7-A we can still get most of the value of running thetests using the same qemu-arm that we used for Armv7-A by building andrunning the test cases for Armv7-A but using the builtins compiled forArmv6-M, Armv7-M or Armv7E-M. This will test that the builtins can be linkedinto a binary and execute the tests correctly but it will not catch if thebuiltins use instructions that are supported on Armv7-A but not Armv6-M,Armv7-M and Armv7E-M.

To get the cmake compile test to pass you will need to pass the librariesneeded to successfully link the cmake test via CMAKE_CFLAGS. It isstrongly recommended that you use version 3.6 or above of cmake so you can useCMAKE_TRY_COMPILE_TARGET=STATIC_LIBRARY to skip the link step.

  • -DCMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY
  • -DCOMPILER_RT_OS_DIR="baremetal"
  • -DCOMPILER_RT_BUILD_BUILTINS=ON
  • -DCOMPILER_RT_BUILD_SANITIZERS=OFF
  • -DCOMPILER_RT_BUILD_XRAY=OFF
  • -DCOMPILER_RT_BUILD_LIBFUZZER=OFF
  • -DCOMPILER_RT_BUILD_PROFILE=OFF
  • -DCMAKE_C_COMPILER=${host_install_dir}/bin/clang
  • -DCMAKE_C_COMPILER_TARGET="your *-none-eabi target"
  • -DCMAKE_ASM_COMPILER_TARGET="your *-none-eabi target"
  • -DCMAKE_AR=/path/to/llvm-ar
  • -DCMAKE_NM=/path/to/llvm-nm
  • -DCMAKE_RANLIB=/path/to/llvm-ranlib
  • -DCOMPILER_RT_BAREMETAL_BUILD=ON
  • -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON
  • -DLLVM_CONFIG_PATH=/path/to/llvm-config
  • -DCMAKE_C_FLAGS="build-c-flags"
  • -DCMAKE_ASM_FLAGS="build-c-flags"
  • -DCOMPILER_RT_EMULATOR="qemu-arm -L /path/to/armv7-A/sysroot"
  • -DCOMPILER_RT_INCLUDE_TESTS=ON
  • -DCOMPILER_RT_TEST_COMPILER="/path/to/clang"
  • -DCOMPILER_RT_TEST_COMPILER_CFLAGS="test-c-flags"

The Armv6-M builtins will use the soft-float ABI. When compiling the tests forArmv7-A we must include "-mthumb -mfloat-abi=soft -mfpu=none" in thetest-c-flags. We must use an Armv7-A soft-float abi sysroot for qemu-arm.

Depending on the linker used for the test cases you may encounter BuildAttributemismatches between the M-profile objects from compiler-rt and the A-profileobjects from the test. The lld linker does not check the profileBuildAttribute so it can be used to link the tests by adding -fuse-ld=lld to theCOMPILER_RT_TEST_COMPILER_CFLAGS.

Alternative using a cmake cache

If you wish to build, but not test compiler-rt for Armv6-M, Armv7-M or Armv7E-Mthe easiest way is to use the BaremetalARM.cmake recipe in clang/cmake/caches.

You will need a bare metal sysroot such as that provided by the GNU ARMEmbedded toolchain.

The libraries can be built with the cmake options:

  • -DBAREMETAL_ARMV6M_SYSROOT=/path/to/bare/metal/toolchain/arm-none-eabi
  • -DBAREMETAL_ARMV7M_SYSROOT=/path/to/bare/metal/toolchain/arm-none-eabi
  • -DBAREMETAL_ARMV7EM_SYSROOT=/path/to/bare/metal/toolchain/arm-none-eabi
  • -C /path/to/llvm/source/tools/clang/cmake/caches/BaremetalARM.cmake
  • /path/to/llvm

Note that for the recipe to work the compiler-rt source must be checked outinto the directory llvm/runtimes. You will also need clang and lld checked out.