Building the Code¶
As discussed in the previous section Project Structure, the algorithms that are provided by this project are implemented in C++ to ensure maximum efficiency (the minimum C++ version required for compilation is specified in the file version-cpp). In addition, a Python wrapper that integrates with the scikit-learn framework is provided (the supported Python versions are specified in the file version-python). To make the underlying C++ implementation accessible from within the Python code, Cython is used.
Unlike pure Python programs, the C++ and Cython source files must be compiled for a particular target platform. To ease the process of compiling the source code, the project comes with a build system that automates the necessary steps. In the following, we discuss the individual steps that are necessary for building the project from scratch. This is necessary if you intend to modify the library’s source code. If you want to use the algorithm without any custom modifications, the installation of pre-built packages is usually a better choice.
Prerequisites¶
As a prerequisite, a supported version of Python, a suitable C++ compiler, as well as optional libraries for multi-threading and GPU support, must be available on the host system. The installation of these software components depends on the operating system at hand. In the following, we provide installation instructions for the supported platforms.
Tip
This project uses Meson as a build system for compiling C++ code. If available on the system, Meson automatically utilizes Ccache for caching previous compilations and detecting when the same compilation is being done again. Compared to the runtime without Ccache, where changes are only detected at the level of entire files, usage of this compiler cache can significantly speed up recompilation and therefore is strongly advised.
Dependency |
Optional |
Description |
|---|---|---|
Python |
No |
Nowadays, most Linux distributions include a pre-installed version of Python 3. If this is not the case, instructions on how to install a recent Python version can be found in Python’s documentation. As noted in this guide, Python should be installed via the distribution’s package manager if possible. |
C++ compiler |
No |
Most Linux distributions provide the GNU Compiler Collection (GCC), which includes a C++ compiler, as part of their software repositories. If this is the case, it can be installed via the distribution’s package manager. |
GoogleTest |
Yes |
The GoogleTest framework must optionally be available in order to compile the project with testing support enabled. It should be possible to install it via the package manager of your Linux distribution. |
OpenMP |
Yes |
OpenMP, which is optionally required for multi-threading support, should be installable via your Linux distribution’s package manager. |
xsimd |
Yes |
To compile the project with SIMD support enabled, version 14 or newer of the library xsimd must be available. On Linux, it can usually be installed via your distribution’s package manager. If unavailable, the build system pulls this particular dependency automatically. |
OpenCL |
Yes |
If the project should be compiled with GPU support, OpenCL must be available. On Linux, it should be installable via your distribution’s package manager. |
Dependency |
Optional |
Description |
|---|---|---|
Python |
No |
Recent versions of macOS do not include Python by default. A suitable Python version can manually be downloaded from the project’s website. Alternatively, the package manager Homebrew can be used for installation via the command |
C++ compiler |
No |
macOS relies on the Clang compiler for building C++ code. It is part of the Xcode developer toolset. |
GoogleTest |
Yes |
The GoogleTest framework must optionally be installed in order to compile the project with testing support enabled. It can easily be installed via Homebrew by running the command |
OpenMP |
Yes |
If the project should be compiled with multi-threading support enabled, the OpenMP library must be installed. We recommend to install it via Homebrew by running the command |
xsimd |
Yes |
To compile the project with SIMD support enabled, the library xsimd must be available. On macOS, we recommend Homebrew to install it via the command |
OpenCL |
Yes |
The Xcode developer toolset should include OpenCL, which are needed for GPU support. However, the OpenCL C++ headers must be installed manually. The easiest way to do so is via the Homebrew command |
Dependency |
Optional |
Description |
|---|---|---|
Python |
No |
Python releases for Windows are available at the project’s website, where you can download an installer. |
C++ compiler |
No |
For the compilation of the project’s source code, the MSVC compiler must be used. It is included in Visual Studio. |
GoogleTest |
Yes |
The GoogleTest framework must optionally be available on your system to compile the project with testing support enabled. It should already be included in recent versions of Visual Studio. |
OpenMP |
Yes |
The Build Tools for Visual Studio also include the OpenMP library, which is utilized by the project for multi-threading support. |
xsimd |
Yes |
To compile the project with SIMD support enabled, the library xsimd must be available. On Windows, we recommend to install the package |
OpenCL |
Yes |
If you intend to compile the project with GPU support enabled, OpenCL must be installed manually. In order to do so, we recommend to install the package |
Additional build- or run-time dependencies are automatically installed when following the instructions below and must not be installed manually.
Tip
Instead of following the instructions below step by step, the following command, which automatically executes all necessary steps, can be used for simplicity.
./build
./build
build.bat
Whenever any C++, Cython or Python source files have been modified, the above command must be run again in order to rebuild modified files and install updated wheel packages into the virtual environment. If any compilation files already exist, this does only result in the affected parts of the code to be rebuilt.
Note
As shown in the section Project Structure, this project is organized in terms of several subprojects. By default, all of these subprojects are built when following the instructions below. However, the environment variable SUBPROJECTS may be used to restrict individual steps of the build process, such as the compilation of C++ and Cython code, the assemblage of Python packages, and the generation of API documentations, to a subset of the available subprojects. As shown below, multiple subprojects can be specified as a comma-separated list:
SUBPROJECTS=common,boosting ./build
SUBPROJECTS=common,boosting ./build
$env:SUBPROJECTS = "common,boosting"
build.bat
Creating a Virtual Environment¶
The build process is based on a virtual Python environment that allows to install build- and run-time dependencies in an isolated manner and independently of the host system. Once the build process was completed, the resulting Python packages are installed into the virtual environment. To create new virtual environment the following command must be executed:
./build venv
./build venv
build.bat venv
Afterward, all necessarily run-time dependencies can be installed:
./build install_runtime_dependencies
./build install_runtime_dependencies
build.bat install_runtime_dependencies
All run-time dependencies (numpy, scipy, etc.) that are required for running the algorithms that are provided by the project should automatically be installed into the virtual environment when executing the above command. As a result, a subdirectory ../../../.venv should have been created in the project’s root directory.
Compiling the C++ Code¶
Once a new virtual environment has successfully been created, the compilation of the C++ code can be started by executing the following command:
./build compile_cpp
./build compile_cpp
build.bat compile_cpp
The compilation is based on the build system Meson and uses Ninja as a backend. After the above command has terminated, a new directory ../../../cpp/build should have been created. It contains the shared libraries (“libmlrlcommon”, “libmlrlboosting” and possibly others) that provide the basic functionality of the project’s algorithms.
Compiling the Cython Code¶
Once the compilation of the C++ code has completed, the Cython code, which allows to access the corresponding shared libraries from within Python, can be compiled in the next step. Again, Meson and Ninja are used for compilation. It can be started via the following command:
./build compile_cython
./build compile_cython
build.bat compile_cython
As a result of executing the above command, the directory ../../../python/build should have been created. It contains Python extension modules for the respective target platform.
Note
Instead of performing the previous steps one after the other, the build target compile can be specified instead of compile_cpp and compile_cython to build the C++ and Cython source files in a single step.
Installing Extension Modules¶
Similar to the previous step, the Python extension modules that have been built from the project’s Cython code must be copied into the Python source tree via the following command:
./build install_cython
./build install_cython
build.bat install_cython
As a result, the compilation files that can be found in the ../../../python/build directories should have been copied into the cython/ subdirectories of each Python module.
Note
Instead of executing the above commands one after the other, the build target install can be used instead of install_cpp and install_cython to copy both, the shared libraries and the extension modules, into the source tree.
Generating pyproject.toml files¶
Once the compilation files have been copied into the Python source tree, pyproject.toml files that provide meta-data for the project’s individual Python modules must be generated via the following command:
./build pyproject_toml
./build pyproject_toml
build.bat pyproject_toml
As a result of running the above command, pyproject.toml files should have been created inside the directories that correspond to the individual Python modules (e.g., in the directory ../../../python/subprojects/common).
Building Wheel Packages¶
Based on the pyproject.toml files generated in the previous step, wheel packages can be built for the individual Python modules via the following command:
./build build_wheels
./build build_wheels
build.bat build_wheels
This should result in .whl files being created in a new dist/ subdirectory inside the directories that correspond to the individual Python modules (e.g., in the directory ../../../python/subprojects/common/dist).
Installing the Wheel Packages¶
The wheel packages that have previously been created can finally be installed into the virtual environment via the following command:
./build install_wheels
./build install_wheels
build.bat install_wheels
After this final step has completed, the Python packages can be used from within the virtual environment once it has been activated. To ensure that the installation of the wheel packages was successful, check if a mlrl/ directory has been created in the lib/ directory of the virtual environment (depending on the Python version and operating system, it should be located at .venv/lib/python3.10/site-packages/mlrl/ or similar). If this is the case, the algorithm can be used from within your own Python code. Alternatively, a command line API can be used to start an experiment (see Using the Command Line API).
Cleaning up Build Files¶
It is possible to delete the compilation files that result from an individual step of the build process mentioned above by using the command line argument --clean or -c. This may be useful if you want to repeat a single or multiple steps of the build process from scratch in case anything went wrong. For example, to delete the C++ compilation files, the following command can be used:
./build --clean compile_cpp
./build --clean compile_cpp
build.bat --clean compile_cpp
If you want to delete all compilation files that have previously been created, including the virtual environment, you should use the following command, where no build target is specified:
./build --clean
./build --clean
build.bat --clean