Handle sources in packages

Handle sources in packages

In the previous tutorial section we created a Conan package for a “Hello World” C++ library. We used the exports_sources attribute of the Conanfile to declare the location of the sources for the library. This method is the simplest way to define the location of the source files when they are in the same folder as the Conanfile. However, sometimes the source files are stored in a repository or a file in a remote server, and not in the same location as the Conanfile. In this section, we will modify the recipe we created previously by adding a source() method and explain how to:

Retrieve the sources from a zip file stored in a remote repository.
Retrieve the sources from a branch of a git repository.

Please, first clone the sources to recreate this project. You can find them in the examples2 repository on GitHub:

$ git clone https://github.com/conan-io/examples2.git
$ cd examples2/tutorial/creating_packages/handle_sources

The structure of the project is the same as the one in the previous example but without the library sources:

.
├── CMakeLists.txt
├── conanfile.py
└── test_package
    ├── CMakeLists.txt
    ├── conanfile.py
    └── src
        └── example.cpp

Sources from a zip file stored in a remote repository

Let’s have a look at the changes in the conanfile.py:

from conan import ConanFile
from conan.tools.cmake import CMakeToolchain, CMake, cmake_layout
from conan.tools.files import get
class helloRecipe(ConanFile):
    name = "hello"
    version = "1.0"
    ...
    # Binary configuration
    settings = "os", "compiler", "build_type", "arch"
    options = {"shared": [True, False], "fPIC": [True, False]}
    default_options = {"shared": False, "fPIC": True}
    def source(self):
        # Please, be aware that using the head of the branch instead of an immutable tag
        # or commit is a bad practice and not allowed by Conan
        get(self, "https://github.com/conan-io/libhello/archive/refs/heads/main.zip",
                  strip_root=True)
    def config_options(self):
        if self.settings.os == "Windows":
            del self.options.fPIC
    def layout(self):
        cmake_layout(self)
    def generate(self):
        tc = CMakeToolchain(self)
        tc.generate()
    def build(self):
        cmake = CMake(self)
        cmake.configure()
        cmake.build()
    def package(self):
        cmake = CMake(self)
        cmake.install()
    def package_info(self):
        self.cpp_info.libs = ["hello"]

As you can see, the recipe is the same but instead of declaring the exports_sources attribute as we did previously, i.e.

exports_sources = "CMakeLists.txt", "src/*", "include/*"

we declare a source() method with this information:

def source(self):
    # Please, be aware that using the head of the branch instead of an immutable tag
    # or commit is strongly discouraged, unsupported by Conan and likely to cause issues
    get(self, "https://github.com/conan-io/libhello/archive/refs/heads/main.zip",
              strip_root=True)

We used the conan.tools.files.get() tool that will first download the zip file from the URL that we pass as an argument and then unzip it. Note that we pass the strip_root=True argument so that if all the unzipped contents are in a single folder, all the contents are moved to the parent folder (check the conan.tools.files.unzip() reference for more details).

Warning

It is expected that retrieving the sources in the future produces the same results. Using mutable source origins, like a moving reference in git (e.g HEAD branch), or the URL to a file where the contents may change over time, is strongly discouraged and not supported. Not following this practice will result in undefined behavior likely to cause breakages

The contents of the zip file are the same as the sources we previously had beside the Conan recipe, so if you do a conan create the results will be the same as before.

$ conan create .
...
-------- Installing packages ----------
Installing (downloading, building) binaries...
hello/1.0: Calling source() in /Users/user/.conan2/p/0fcb5ffd11025446/s/.
Downloading update_source.zip
hello/1.0: Unzipping 3.7KB
Unzipping 100 %
hello/1.0: Copying sources to build folder
hello/1.0: Building your package in /Users/user/.conan2/p/tmp/369786d0fb355069/b
...
-------- Testing the package: Running test() ----------
hello/1.0 (test package): Running test()
hello/1.0 (test package): RUN: ./example
hello/1.0: Hello World Release!
hello/1.0: __x86_64__ defined
hello/1.0: __cplusplus199711
hello/1.0: __GNUC__4
hello/1.0: __GNUC_MINOR__2
hello/1.0: __clang_major__13
hello/1.0: __clang_minor__1
hello/1.0: __apple_build_version__13160021

Please, check the highlighted lines with the messages about the download and unzip operation.

Sources from a branch in a git repository

Now, let’s modify the source() method to bring the sources from a git repository instead of a zip file. We show just the relevant parts:

...
from conan.tools.scm import Git
class helloRecipe(ConanFile):
    name = "hello"
    version = "1.0"
    ...
    def source(self):
        git = Git(self)
        git.clone(url="https://github.com/conan-io/libhello.git", target=".")
    ...

Here, we use the conan.tools.scm.Git() tool. The Git class implements several methods to work with git repositories. In this case, we call the clone method to clone the https://github.com/conan-io/libhello.git repository in the default branch using the same folder for cloning the sources instead of a subfolder (passing the target="." argument).

Warning

As above, this is only a simple example. The source origin for Git() also has to be immutable, it is necessary to checkout out an immutable tag or a specific commit to guarantee the correct behavior. Using the HEAD of the repository is not allowed and can cause undefined behavior and breakages.

To checkout a commit or tag in the repository we use the checkout() method of the Git tool:

def source(self):
    git = Git(self)
    git.clone(url="https://github.com/conan-io/libhello.git", target=".")
    git.checkout("<tag> or <commit hash>")

For more information about the Git class methods, please check the conan.tools.scm.Git() reference.

Note that it’s also possible to run other commands by invoking the self.run() method.

Using the conandata.yml file

We can write a file named conandata.yml in the same folder of the conanfile.py. This file will be automatically exported and parsed by Conan and we can read that information from the recipe. This is handy for example to extract the URLs of the external sources repositories, zip files etc. This is an example of conandata.yml:

sources:
  "1.0":
    url: "https://github.com/conan-io/libhello/archive/refs/heads/main.zip"
    sha256: "7bc71c682895758a996ccf33b70b91611f51252832b01ef3b4675371510ee466"
    strip_root: true
  "1.1":
    url: ...
    sha256: ...

The recipe doesn’t need to be modified for each version of the code. We can pass all the keys of the specified version (url, sha256, and strip_root) as arguments to the get function, that, in this case, allow us to verify that the downloaded zip file has the correct sha256. So we could modify the source method to this:

def source(self):
    get(self, **self.conan_data["sources"][self.version])
    # Similar to:
    # data = self.conan_data["sources"][self.version]
    # get(self, data["url"], sha256=data["sha256"], strip_root=data["strip_root"])