Benchmark app code

The Jetpack Benchmark library allows you to quickly benchmark your Kotlin-basedor Java-based code from within Android Studio. The library handles warmup,measures your code performance, and outputs benchmarking results to the AndroidStudio console.

Use cases include scrolling a RecyclerView, inflating a non-trivial Viewhierarchy, and performing database queries.

If you haven't yet adopted AndroidX in a project you want to benchmark, seeMigrate an existing project using Android Studio.

Quickstart

This section provides a quick set of steps to try out benchmarking withoutrequiring you to move code into modules. Because the steps involvedisabling debugging for accurate performance results, you won't committhe changes to your source control system, but it can still be helpful when youwant to run one-off measurements.

To quickly perform one-off benchmarking, do the following:

  • Add the library to your module’s build.gradle file:

project_root/module_dir/build.gradle

  1. dependencies {
  2.     androidTestImplementation "androidx.benchmark:benchmark:1.0.0-alpha01"
  3. }
  • To disable debugging in the test manifest, update your <application>element to force-disable debugging temporarily as follows:

project_root/module_dir/src/androidTest/AndroidManifest.xml

  1. <!-- Important: disable debuggable for accurate performance results -->
  2. <application
  3.     android:debuggable="false"
  4.     tools:ignore="HardcodedDebugMode"
  5.     tools:replace="android:debuggable"/>
  • To add your benchmark, add an instance of BenchmarkRule in a testfile in the androidTest directory. For more information on writingbenchmarks, see Write a benchmark.

The following code snippet shows how to add a benchmark to a JUnit test:

Kotlin

  1. @RunWith(AndroidJUnit4::class)
    class MyBenchmark {
        @get:Rule
        val benchmarkRule = BenchmarkRule()
  2. @Test
  3. fun benchmarkSomeWork() = benchmarkRule.measureRepeated {
  4.     doSomeWork()
  5. }
  6. }

Java

  1. @RunWith(AndroidJUnit4.class)
    class MyBenchmark {
        @Rule
        public BenchmarkRule benchmarkRule = new BenchmarkRule();
  2. @Test
  3. public void myBenchmark() {
  4.     final BenchmarkState state = benchmarkRule.getState();
  5.     while (state.keepRunning()) {
  6.         doSomeWork();
  7.     }
  8. }
  9. }

What to benchmark

Benchmarks are most useful for CPU work that is run many times in your app. Goodexamples are RecyclerView scrolling, data conversions/processing, and piecesof code that get used repeatedly.

Other types of code are more difficult to measure with benchmarking. Becausebenchmarks run in a loop, any code that isn't run frequently, or performsdifferently when called multiple times, may not be a good fit for benchmarking.

Caching

Try to avoid measuring just the cache. For example, a custom view'slayout benchmark might measure only the performance of the layout cache. Toavoid this, you can pass different layout parameters in each loop. In othercases, such as when measuring file system performance, this may be difficultbecause the OS caches the file system while in a loop.

Infrequently-run code

Code that's run once during application startup is not very likely to get JITcompiled by Android Runtime (ART). Because of that, benchmarking this code as itruns in a loop isn't a realistic way to measure its performance.

For this sort of code, we recommend tracing or profiling the code inyour app, instead. Note that this doesn't mean you can't benchmark code inyour startup path, but rather that you should pick code that's run in a loop,and that is likely to get JIT compiled.

Full project setup

To set up benchmarking for regular benchmarking rather than one-offbenchmarking, you isolate benchmarks into their own module. This ensures thattheir configuration, such as setting debuggable to false, is separatefrom regular tests.

To do this, you need to complete these tasks:

  • Place code and resources you want to benchmark into a library module if theyaren't already in one.

  • Add a new library module to hold the benchmarks themselves.

Our samples give examples of how to set up a project in this way.

Set Android Studio properties

Jetpack Benchmark library is currently in Alpha, and requires manuallysetting Android Studio properties to enable benchmark module wizard support.

To enable the Android Studio template for benchmarking, do the following:

npw.benchmark.template.module=true

  • Save and close the file.

  • Restart Android Studio.

Create a new module

The benchmarking module template automatically configures settings forbenchmarking.

To use the module template to create a new module, do the following:

  • Right-click your project or module and select New > Module.

  • Select Benchmark Module and click Next.

Benchmark your app - 图1

Figure 1. Benchmark module

  • Enter a module name, choose the language, and click Finish.

A module is created that is pre-configured for benchmarking, with a benchmark directory added and debuggable set to false.

Note: debuggable=false prevents using the debugger and profiling toolswith your tests. Using a separate library module eliminates this drawback.

Write a benchmark

Benchmarks are standard instrumentation tests. To create a benchmark, use theBenchmarkRule class provided by the library. To benchmark activities, useActivityTestRule or ActivityScenarioRule. To benchmark UI code,use @UiThreadTest.

The following code shows a sample benchmark:

Kotlin

  1. @RunWith(AndroidJUnit4::class)
    class ViewBenchmark {
        @get:Rule
        val benchmarkRule = BenchmarkRule()
  2. @Test
  3. fun simpleViewInflate() {
  4.     val context = ApplicationProvider.getApplicationContext<context>()
  5.     val inflater = LayoutInflater.from(context)
  6.     val root = FrameLayout(context)
  8.     benchmarkRule.keepRunning {
  9.         inflater.inflate(R.layout.test_simple_view, root, false)
  10.     }
  11. }
  12. }

Java

  1. @RunWith(AndroidJUnit4::class)
    public class ViewBenchmark {
        @Rule
        public BenchmarkRule benchmarkRule = new BenchmarkRule();
  2. @Test
  3. public void simpleViewInflate() {
  4.     Context context = ApplicationProvider.getApplicationContext<context>();
  5.     final BenchmarkState state = benchmarkRule.getState();
  6.     LayoutInflater inflater = LayoutInflater.from(context);
  7.     FrameLayout root = new FrameLayout(context);
  9.     while (state.keepRunning()) {
  10.         inflater.inflate(R.layout.test_simple_view, root, false);
  11.     }
  12. }
  13. }

You can disable timing for sections of code you don't want to measure, as shownin the following code sample:

Kotlin

  1. @Test
    fun bitmapProcessing() = benchmarkRule.measureRepeated {
        val input: Bitmap = runWithTimingDisabled { constructTestBitmap() }
        processBitmap(input)
    }

Java

  1. @Test
    public void bitmapProcessing() {
        final BenchmarkState state = benchmarkRule.getState();
        while (state.keepRunning()) {
            state.pauseTiming();
            Bitmap input = constructTestBitmap();
            state.resumeTiming();
  2.     processBitmap(input);
  3. }
  4. }

For information on running the benchmark, seeRun the benchmark.

Run the benchmark

In Android Studio, the benchmark will run when you run your app. On AndroidStudio 3.4 and higher, you can see output sent to the console.

To run the benchmark, in the module, navigate to benchmark/src/androidTest andpress Control+Shift+F10 (Command+Shift+R on Mac). Results of thebenchmark appear in the console, as shown in Figure 2:

Benchmarking output in Android Studio

Figure 2. Benchmarking output in Android Studio

From the command line, run the regular connectedCheck:

  1. ./gradlew benchmark:connectedCheck

Collect the data

In addition to sending output to the console in Android Studio, the benchmarkresults are written to disk in both JSON and XML formats. These are written tothe external shared downloads folder on the device (usually at/storage/emulated/0/Download/<app_id>-benchmarkData.<extension>) and on thehost at<project_root>/<module>/build/benchmark_reports/<app_id>-benchmarkData.<extension>.

Note: For results to be copied to the host when benchmarks are complete, youmust have configured the androidx.benchmark Gradle plugin, as covered inLock clocks (requires root).

Clock stability

Clocks on mobile devices dynamically change from high state (for performance)to low state (to save power, or when the device gets hot). These varying clockscan make your benchmark numbers vary widely, so the library provides ways todeal with this issue.

Lock clocks (requires root)

Locking clocks is the best way to get stable performance. It ensures that clocksnever get high enough to heat up the device, or low if a benchmark isn't fullyutilizing the CPU. While this is the best way to ensure stable performance, itisn't supported on most devices, due to requiring adb root.

To lock your clocks, add the supplied helper plugin to the top level project’sclasspath in the main build.gradle file:

  1. buildscript {
  2.     ...
  3.     dependencies {
  4.         ...
  5.         classpath "androidx.benchmark:benchmark-gradle-plugin:1.0.0-alpha01"
  6.     }
  7. }

Apply the plugin in the build.gradle of the module you are benchmarking:

  1. apply plugin: com.android.app
  2. apply plugin: androidx.benchmark
  3. ...

This adds benchmarking Gradle tasks to your project, including./gradlew lockClocks and ./gradlew unlockClocks. Use these tasks to lock andunlock a device’s CPU using adb.

If you have multiple devices visible to adb, use the environment variableANDROID_SERIAL to specify which device the Gradle task should operate on:

  1. ANDROID_SERIAL=device-id-from-adb-devices ./gradlew lockClocks

Note: Rebooting resets your device clocks.

Sustained performance mode

Window.setSustainedPerformanceMode()) is a feature supported by somedevices that enables an app to opt for a lower max CPU frequency. When runningon supported devices, the Benchmark library uses a combination of this API andlaunching its own activity to both prevent thermal throttling and stabilizeresults.

This functionality is already enabled in modules created using the benchmarkmodule template. To enable this functionality in other modules, add theinstrumentation runner provided by the library to your benchmark module'sbuild.gradle file:

project_root/benchmark/build.gradle

  1. android {
  2.     defaultConfig {
  3.         testInstrumentationRunner "androidx.benchmark.AndroidBenchmarkRunner"
  4.     }
  5. }

The runner launches an opaque, fullscreen activity to ensure that the benchmarkruns in the foreground and without any other app drawing.

Automatic execution pausing

If neither clock-locking nor sustained performance are used, the libraryperforms automatic thermal throttling detection. When enabled, the internalbenchmark periodically runs to determine when the device temperature has gottenhigh enough to lower CPU performance. When lowered CPU performance is detected,the library pauses execution to let the device cool down, and retries thecurrent benchmark.

Warnings

The library detects the following conditions to ensure your project andenvironment are set up for release-accurate performance:

  • Debuggable is set to false.
  • A physical device, not an emulator, is being used.
  • Clocks are locked if the device is rooted.
    If any of the above checks fails, test names are intentionally corrupted whenreporting results by prepending "DEBUGGABLE_." This is done to discourage useof inaccurate measurements.

Benchmarking samples

Sample benchmark code is available in the following projects:

Provide feedback

To report issues or submit feature requests when using benchmarking, see thepublic issue tracker.