Creating a Conan package using a Docker runner

Warning

This feature is experimental and subject to breaking changes. See the Conan stability section for more information.

In this example we are going to see how to create the zlib/1.3.1 Conan packge inside Docker using a runner. Let’s create two profiles and a Dockerfile inside our project folder.

  1. $ cd </my/runner/folder>
  2. $ tree
  3. .
  4. ├── Dockerfile
  5. ├── docker_example_build
  6. └── docker_example_host

docker_example_host profile

  1. [settings]
  2. arch=x86_64
  3. build_type=Release
  4. compiler=gcc
  5. compiler.cppstd=gnu17
  6. compiler.libcxx=libstdc++11
  7. compiler.version=11
  8. os=Linux
  9. [runner]
  10. type=docker
  11. dockerfile=</my/runner/folder>
  12. cache=copy
  13. remove=true

docker_example_build profile

  1. [settings]
  2. arch=x86_64
  3. build_type=Release
  4. compiler=gcc
  5. compiler.cppstd=gnu17
  6. compiler.libcxx=libstdc++11
  7. compiler.version=11
  8. os=Linux

Dockerfile

  1. FROM ubuntu:22.04
  2. RUN apt-get update \
  3.     && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
  4.         build-essential \
  5.         cmake \
  6.         python3 \
  7.         python3-pip \
  8.         python3-venv \
  9.     && rm -rf /var/lib/apt/lists/*
  10. RUN pip install conan

In this example we are going to start from a totally clean docker, without containers or images. In addition, we are going to have the conan cache also completely empty.

  1. $ conan list "*:*"
  2. Found 0 pkg/version recipes matching * in local cache
  4. $ docker ps --all
  5. CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
  7. $ docker images
  8. REPOSITORY   TAG       IMAGE ID   CREATED   SIZE

Now, we are going to clone and build zlib from conan-center-index and create it using our new runner definition.

  1. $ git clone https://github.com/conan-io/conan-center-index.git --depth=1
  2. $ conan create ./conan-center-index/recipes/zlib/all --version 1.3.1 -pr:h </my/runner/folder>/docker_example_host -pr:b </my/runner/folder>/docker_example_build

If we split and analyze the command output, we can see what is happening and where the commands are being executed.

  1. Standard conan execution.
  1. ======== Exporting recipe to the cache ========
  2. zlib/1.3.1: Exporting package recipe: </my/runner/folder>/conan-center-index/recipes/zlib/all/conanfile.py
  3. zlib/1.3.1: exports: File 'conandata.yml' found. Exporting it...
  4. zlib/1.3.1: Calling export_sources()
  5. zlib/1.3.1: Copied 1 '.py' file: conanfile.py
  6. zlib/1.3.1: Copied 1 '.yml' file: conandata.yml
  7. zlib/1.3.1: Copied 1 '.patch' file: 0001-fix-cmake.patch
  8. zlib/1.3.1: Exported to cache folder: /Users/conan/.conan2/p/zlib95420566fc0dd/e
  9. zlib/1.3.1: Exported: zlib/1.3.1#e20364c96c45455608a72543f3a53133 (2024-04-29 17:03:44 UTC)
  11. ======== Input profiles ========
  12. Profile host:
  13. [settings]
  14. arch=x86_64
  15. build_type=Release
  16. compiler=gcc
  17. compiler.cppstd=gnu17
  18. compiler.libcxx=libstdc++11
  19. compiler.version=11
  20. os=Linux
  22. Profile build:
  23. [settings]
  24. arch=x86_64
  25. build_type=Release
  26. compiler=gcc
  27. compiler.cppstd=gnu17
  28. compiler.libcxx=libstdc++11
  29. compiler.version=11
  30. os=Linux
  1. Build docker image
  1. **********************************************
  2. * Building the Docker image: my-conan-runner *
  3. **********************************************
  5. Dockerfile path: '</my/runner/folder>/Dockerfile'
  6. Docker build context: '</my/runner/folder>'
  8. Step 1/4 : FROM ubuntu:22.04
  10. ...
  12. ---> dba927bb0517
  13. Successfully built dba927bb0517
  14. Successfully tagged my-conan-runner:latest
  1. Save the local cache running conan cache save.
  1. ******************************************************************************************************************
  2. * Save host cache in: </my/runner/folder>/conan-center-index/recipes/zlib/all/.conanrunner/local_cache_save.tgz *
  3. ******************************************************************************************************************
  5. Found 1 pkg/version recipes matching * in local cache
  6. Saving zlib/1.3.1: p/zlib95420566fc0dd
  1. Create and initialize the docker container.
  1. *********************************
  2. * Creating the docker container *
  3. *********************************
  5. *****************************************
  6. * Container conan-runner-docker running *
  7. *****************************************
  1. Check if the container has a conan version with the runner feature.
  1. *******************************************
  2. * Running in container: "conan --version" *
  3. *******************************************
  5. Conan version 2.3.0
  1. Initialize the container conan cache using the host copy running conan cache restore.
  1. *********************************************************************************************************
  2. * Running in container: "conan cache restore "/root/conanrunner/all/.conanrunner/local_cache_save.tgz"" *
  3. *********************************************************************************************************
  5. Restore: zlib/1.3.1 in p/zlib95420566fc0dd
  6. Local Cache
  7. zlib
  8.     zlib/1.3.1
  9.     revisions
  10.         e20364c96c45455608a72543f3a53133 (2024-04-29 17:19:32 UTC)
  11.         packages
  12.         recipe_folder: p/zlib95420566fc0dd
  1. Run the conan create inside the container and build zlib.
  1. *****************************************************************************************************************************************************************************************************************************************************
  2. * Running in container: "conan create /root/conanrunner/all --version 1.3.1 -pr:h /root/conanrunner/all/.conanrunner/profiles/docker_example_host_1 -pr:b /root/conanrunner/all/.conanrunner/profiles/docker_example_build_0 -f json > create.json" *
  3. *****************************************************************************************************************************************************************************************************************************************************
  6. ======== Exporting recipe to the cache ========
  7. zlib/1.3.1: Exporting package recipe: /root/conanrunner/all/conanfile.py
  8. zlib/1.3.1: exports: File 'conandata.yml' found. Exporting it...
  9. zlib/1.3.1: Calling export_sources()
  10. zlib/1.3.1: Copied 1 '.yml' file: conandata.yml
  11. zlib/1.3.1: Copied 1 '.py' file: conanfile.py
  12. zlib/1.3.1: Copied 1 '.patch' file: 0001-fix-cmake.patch
  13. zlib/1.3.1: Exported to cache folder: /root/.conan2/p/zlib95420566fc0dd/e
  14. zlib/1.3.1: Exported: zlib/1.3.1#e20364c96c45455608a72543f3a53133 (2024-04-29 17:19:32 UTC)
  16. ======== Input profiles ========
  17. Profile host:
  18. [settings]
  19. arch=x86_64
  20. build_type=Release
  21. compiler=gcc
  22. compiler.cppstd=gnu17
  23. compiler.libcxx=libstdc++11
  24. compiler.version=11
  25. os=Linux
  27. Profile build:
  28. [settings]
  29. arch=x86_64
  30. build_type=Release
  31. compiler=gcc
  32. compiler.cppstd=gnu17
  33. compiler.libcxx=libstdc++11
  34. compiler.version=11
  35. os=Linux
  38. ======== Computing dependency graph ========
  39. Graph root
  40.     cli
  41. Requirements
  42.     zlib/1.3.1#e20364c96c45455608a72543f3a53133 - Cache
  44. ======== Computing necessary packages ========
  45. zlib/1.3.1: Forced build from source
  46. Requirements
  47.     zlib/1.3.1#e20364c96c45455608a72543f3a53133:b647c43bfefae3f830561ca202b6cfd935b56205 - Build
  49. ======== Installing packages ========
  50. zlib/1.3.1: Calling source() in /root/.conan2/p/zlib95420566fc0dd/s/src
  52. -------- Installing package zlib/1.3.1 (1 of 1) --------
  53. zlib/1.3.1: Building from source
  54. zlib/1.3.1: Package zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205
  55. zlib/1.3.1: Copying sources to build folder
  56. zlib/1.3.1: Building your package in /root/.conan2/p/b/zlib8dd8e27348e8c/b
  57. zlib/1.3.1: Calling generate()
  58. zlib/1.3.1: Generators folder: /root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release/generators
  59. zlib/1.3.1: CMakeToolchain generated: conan_toolchain.cmake
  60. zlib/1.3.1: CMakeToolchain generated: /root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release/generators/CMakePresets.json
  61. zlib/1.3.1: CMakeToolchain generated: /root/.conan2/p/b/zlib8dd8e27348e8c/b/src/CMakeUserPresets.json
  62. zlib/1.3.1: Generating aggregated env files
  63. zlib/1.3.1: Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
  64. zlib/1.3.1: Calling build()
  65. zlib/1.3.1: Apply patch (conan): separate static/shared builds, disable debug suffix
  66. zlib/1.3.1: Running CMake.configure()
  67. zlib/1.3.1: RUN: cmake -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE="generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="/root/.conan2/p/b/zlib8dd8e27348e8c/p" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Release" "/root/.conan2/p/b/zlib8dd8e27348e8c/b/src"
  68. -- Using Conan toolchain: /root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release/generators/conan_toolchain.cmake
  69. -- Conan toolchain: Setting CMAKE_POSITION_INDEPENDENT_CODE=ON (options.fPIC)
  70. -- Conan toolchain: Setting BUILD_SHARED_LIBS = OFF
  71. -- The C compiler identification is GNU 11.4.0
  72. -- Detecting C compiler ABI info
  73. -- Detecting C compiler ABI info - done
  74. -- Check for working C compiler: /usr/bin/cc - skipped
  75. -- Detecting C compile features
  76. -- Detecting C compile features - done
  77. -- Looking for sys/types.h
  78. -- Looking for sys/types.h - found
  79. -- Looking for stdint.h
  80. -- Looking for stdint.h - found
  81. -- Looking for stddef.h
  82. -- Looking for stddef.h - found
  83. -- Check size of off64_t
  84. -- Check size of off64_t - done
  85. -- Looking for fseeko
  86. -- Looking for fseeko - found
  87. -- Looking for unistd.h
  88. -- Looking for unistd.h - found
  89. -- Renaming
  90. --     /root/.conan2/p/b/zlib8dd8e27348e8c/b/src/zconf.h
  91. -- to 'zconf.h.included' because this file is included with zlib
  92. -- but CMake generates it automatically in the build directory.
  93. -- Configuring done
  94. -- Generating done
  95. -- Build files have been written to: /root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release
  96. zlib/1.3.1: Running CMake.build()
  97. zlib/1.3.1: RUN: cmake --build "/root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release" -- -j16
  98. [ 12%] Building C object CMakeFiles/zlib.dir/adler32.c.o
  99. [ 12%] Building C object CMakeFiles/zlib.dir/compress.c.o
  100. [ 18%] Building C object CMakeFiles/zlib.dir/deflate.c.o
  101. [ 25%] Building C object CMakeFiles/zlib.dir/crc32.c.o
  102. [ 31%] Building C object CMakeFiles/zlib.dir/gzlib.c.o
  103. [ 37%] Building C object CMakeFiles/zlib.dir/gzread.c.o
  104. [ 43%] Building C object CMakeFiles/zlib.dir/gzclose.c.o
  105. [ 56%] Building C object CMakeFiles/zlib.dir/infback.c.o
  106. [ 56%] Building C object CMakeFiles/zlib.dir/gzwrite.c.o
  107. [ 62%] Building C object CMakeFiles/zlib.dir/inflate.c.o
  108. [ 68%] Building C object CMakeFiles/zlib.dir/inffast.c.o
  109. [ 75%] Building C object CMakeFiles/zlib.dir/trees.c.o
  110. [ 81%] Building C object CMakeFiles/zlib.dir/zutil.c.o
  111. [ 87%] Building C object CMakeFiles/zlib.dir/uncompr.c.o
  112. [ 93%] Building C object CMakeFiles/zlib.dir/inftrees.c.o
  113. [100%] Linking C static library libz.a
  114. [100%] Built target zlib
  115. zlib/1.3.1: Package 'b647c43bfefae3f830561ca202b6cfd935b56205' built
  116. zlib/1.3.1: Build folder /root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release
  117. zlib/1.3.1: Generating the package
  118. zlib/1.3.1: Packaging in folder /root/.conan2/p/b/zlib8dd8e27348e8c/p
  119. zlib/1.3.1: Calling package()
  120. zlib/1.3.1: Running CMake.install()
  121. zlib/1.3.1: RUN: cmake --install "/root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release" --prefix "/root/.conan2/p/b/zlib8dd8e27348e8c/p"
  122. -- Install configuration: "Release"
  123. -- Installing: /root/.conan2/p/b/zlib8dd8e27348e8c/p/lib/libz.a
  124. -- Installing: /root/.conan2/p/b/zlib8dd8e27348e8c/p/include/zconf.h
  125. -- Installing: /root/.conan2/p/b/zlib8dd8e27348e8c/p/include/zlib.h
  127. zlib/1.3.1: package(): Packaged 1 file: LICENSE
  128. zlib/1.3.1: package(): Packaged 2 '.h' files: zlib.h, zconf.h
  129. zlib/1.3.1: package(): Packaged 1 '.a' file: libz.a
  130. zlib/1.3.1: Created package revision fd85b1346d5377ae2465645768e62bf2
  131. zlib/1.3.1: Package 'b647c43bfefae3f830561ca202b6cfd935b56205' created
  132. zlib/1.3.1: Full package reference: zlib/1.3.1#e20364c96c45455608a72543f3a53133:b647c43bfefae3f830561ca202b6cfd935b56205#fd85b1346d5377ae2465645768e62bf2
  133. zlib/1.3.1: Package folder /root/.conan2/p/b/zlib8dd8e27348e8c/p
  134. WARN: deprecated: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X:
  135. WARN: deprecated:     'cpp_info.names' used in: zlib/1.3.1
  137. ======== Launching test_package ========
  139. ======== Computing dependency graph ========
  140. Graph root
  141.     zlib/1.3.1 (test package): /root/conanrunner/all/test_package/conanfile.py
  142. Requirements
  143.     zlib/1.3.1#e20364c96c45455608a72543f3a53133 - Cache
  145. ======== Computing necessary packages ========
  146. Requirements
  147.     zlib/1.3.1#e20364c96c45455608a72543f3a53133:b647c43bfefae3f830561ca202b6cfd935b56205#fd85b1346d5377ae2465645768e62bf2 - Cache
  149. ======== Installing packages ========
  150. zlib/1.3.1: Already installed! (1 of 1)
  151. WARN: deprecated: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X:
  152. WARN: deprecated:     'cpp_info.names' used in: zlib/1.3.1
  154. ======== Testing the package ========
  155. Removing previously existing 'test_package' build folder: /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release
  156. zlib/1.3.1 (test package): Test package build: build/gcc-11-x86_64-gnu17-release
  157. zlib/1.3.1 (test package): Test package build folder: /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release
  158. zlib/1.3.1 (test package): Writing generators to /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release/generators
  159. zlib/1.3.1 (test package): Generator 'CMakeToolchain' calling 'generate()'
  160. zlib/1.3.1 (test package): CMakeToolchain generated: conan_toolchain.cmake
  161. zlib/1.3.1 (test package): CMakeToolchain generated: /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release/generators/CMakePresets.json
  162. zlib/1.3.1 (test package): CMakeToolchain generated: /root/conanrunner/all/test_package/CMakeUserPresets.json
  163. zlib/1.3.1 (test package): Generator 'CMakeDeps' calling 'generate()'
  164. zlib/1.3.1 (test package): CMakeDeps necessary find_package() and targets for your CMakeLists.txt
  165.     find_package(ZLIB)
  166.     target_link_libraries(... ZLIB::ZLIB)
  167. zlib/1.3.1 (test package): Generator 'VirtualRunEnv' calling 'generate()'
  168. zlib/1.3.1 (test package): Generating aggregated env files
  169. zlib/1.3.1 (test package): Generated aggregated env files: ['conanrun.sh', 'conanbuild.sh']
  171. ======== Testing the package: Building ========
  172. zlib/1.3.1 (test package): Calling build()
  173. zlib/1.3.1 (test package): Running CMake.configure()
  174. zlib/1.3.1 (test package): RUN: cmake -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE="generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="/root/conanrunner/all/test_package" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Release" "/root/conanrunner/all/test_package"
  175. -- Using Conan toolchain: /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release/generators/conan_toolchain.cmake
  176. -- Conan toolchain: C++ Standard 17 with extensions ON
  177. -- The C compiler identification is GNU 11.4.0
  178. -- Detecting C compiler ABI info
  179. -- Detecting C compiler ABI info - done
  180. -- Check for working C compiler: /usr/bin/cc - skipped
  181. -- Detecting C compile features
  182. -- Detecting C compile features - done
  183. -- Conan: Target declared 'ZLIB::ZLIB'
  184. -- Configuring done
  185. -- Generating done
  186. -- Build files have been written to: /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release
  187. zlib/1.3.1 (test package): Running CMake.build()
  188. zlib/1.3.1 (test package): RUN: cmake --build "/root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release" -- -j16
  189. [ 50%] Building C object CMakeFiles/test_package.dir/test_package.c.o
  190. [100%] Linking C executable test_package
  191. [100%] Built target test_package
  193. ======== Testing the package: Executing test ========
  194. zlib/1.3.1 (test package): Running test()
  195. zlib/1.3.1 (test package): RUN: ./test_package
  196. Compressed size is: 21
  197. Compressed string is: Conan Package Manager
  198. Compressed size is: 22
  199. Compressed string is: xsKHLNLOUMRE
  200. ZLIB VERSION: 1.3.1
  1. Copy just the package created inside the container using the pkglist.json info from the previous conan create, restore this new package inside the host cache running a conan cache save and remove the container.
  1. **********************************************************************************************************************************
  2. * Running in container: "conan cache save --list=pkglist.json --file "/root/conanrunner/all"/.conanrunner/docker_cache_save.tgz" *
  3. **********************************************************************************************************************************
  5. Saving zlib/1.3.1: p/zlib95420566fc0dd
  6. Saving zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205: p/b/zlib8dd8e27348e8c/p
  7. Saving zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 metadata: p/b/zlib8dd8e27348e8c/d/metadata
  8. Local Cache
  9. zlib
  10.     zlib/1.3.1
  11.     revisions
  12.         e20364c96c45455608a72543f3a53133 (2024-04-29 17:19:32 UTC)
  13.         packages
  14.             b647c43bfefae3f830561ca202b6cfd935b56205
  15.             revisions
  16.                 fd85b1346d5377ae2465645768e62bf2
  17.                 package_folder: p/b/zlib8dd8e27348e8c/p
  18.                 metadata_folder: p/b/zlib8dd8e27348e8c/d/metadata
  19.             info
  20.                 settings
  21.                 os: Linux
  22.                 arch: x86_64
  23.                 compiler: gcc
  24.                 compiler.version: 11
  25.                 build_type: Release
  26.                 options
  27.                 fPIC: True
  28.                 shared: False
  29.         recipe_folder: p/zlib95420566fc0dd
  32. ************************************************************************************************************************
  33. * Restore host cache from: </my/runner/folder>/conan-center-index/recipes/zlib/all/.conanrunner/docker_cache_save.tgz *
  34. ************************************************************************************************************************
  36. Restore: zlib/1.3.1 in p/zlib95420566fc0dd
  37. Restore: zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 in p/b/zlib8dd8e27348e8c/p
  38. Restore: zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 metadata in p/b/zlib8dd8e27348e8c/d/metadata
  40. **********************
  41. * Stopping container *
  42. **********************
  45. **********************
  46. * Removing container *
  47. **********************

If we now check the status of our conan and docker cache, we will see the new zlib package compile for Linux and the new docker image. We don’t have any container because we define remove=true

  1. $ conan list "*:*"
  2. Found 1 pkg/version recipes matching * in local cache
  3. Local Cache
  4.   zlib
  5.     zlib/1.3.1
  6.     revisions
  7.         e20364c96c45455608a72543f3a53133 (2024-04-29 17:18:07 UTC)
  8.         packages
  9.             b647c43bfefae3f830561ca202b6cfd935b56205
  10.             info
  11.                 settings
  12.                 arch: x86_64
  13.                 build_type: Release
  14.                 compiler: gcc
  15.                 compiler.version: 11
  16.                 os: Linux
  17.                 options
  18.                 fPIC: True
  19.                 shared: False
  21. $ docker ps --all
  22. CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
  24. $ docker images
  25. REPOSITORY        TAG       IMAGE ID       CREATED          SIZE
  26. my-conan-runner   latest    383b905f352e   22 minutes ago   531MB
  27. ubuntu            22.04     437ec753bef3   12 days ago      77.9MB