About the Benchmarking Process

What’s the difference between the four images?

In the benchmarking charts you’ll see four images: Few Large Layers, Few Small Layers, Many Large Layers, and Many Small Layers

All the images use the same base image: gcr.io/buildpacks/builder:v1

Few vs Many

Few will copy two files while many will copy 20 files.

Small vs Large

Small will copy a 20MB file while large will copy a 123MB file.

Using this info you can see the following:

  • Few Large Layers: copies two 123MB files
  • Few Small Layers: copies two 20MB files
  • Many Large Layers: copies 20 123MB files
  • Many Small Layers: copies 20 20MB files

Finally, as the last layer, a simplistic 11 line Go app is copied in.

Iterative vs Initial

There are two graphs for each benchmark, iterative and inital.

Inital

Initial simulates loading the image for the first time.

All existing images and cache is removed/cleared from minikube and Docker between runs to replicate what the user would experience when loading for the first time.

Iterative

Iterative simulates only the Go app (last layer of the image) changing.

This is the exact use case of Skaffold, where if the user changes a file the Go binary is rebuilt and the image is re-loaded.

Bewteen runs the cache and existing image is left alone, only the Go binary is changed.

How are the benchmarks conducted?

  1. // Pseudo code of running docker-env benchmark
  3. startMininkube() // minikube start --container-runtime=docker
  5. for image in [fewLargeLayers, fewSmallLayers, ...] {
  6.     buildGoBinary()
  8.     // inital simulation
  9.     for i in runCount {
  10.         startTimer()
  12.         runDockerEnvImageLoad(image)
  14.         stopTimer()
  16.         verifyImageSuccessfullyLoaded()
  18.         storeTimeTaken()
  20.         removeImage()
  22.         clearDockerCache()
  23.     }
  25.     // iterative simulation
  26.     for i in runCount {
  27.         updateGoBinary()
  29.         startTimer()
  31.         runDockerEnvImageLoad(image)
  33.         stopTimer()
  35.         verifyImageSuccessfullyLoaded()
  37.         storeTimeTaken() // skip if first run
  38.     }
  40.     clearDockerCache()
  42.     calculateAndRecordAverageTime()
  43. }
  45. deleteMinikube() // minkube delete --all

Last modified July 28, 2021: resolve change requests for #12040 (d1cdd24ef)