Building from Source¶
As discussed in the previous section Project Structure, the algorithms that are provided by this project are implemented in C++ to ensure maximum efficiency (requires C++ 17 or newer). In addition, a Python wrapper that integrates with the scikit-learn framework is provided (requires Python 3.9 or newer). To make the underlying C++ implementation accessible from within the Python code, Cython is used (requires Cython 3.0 or newer).
Unlike pure Python programs, the C++ and Cython source files must be compiled for a particular target platform. To ease the process of compiling the source code, the project comes with a SCons build that automates the necessary steps. In the following, we discuss the individual steps that are necessary for building the project from scratch. This is necessary if you intend to modify the library’s source code. If you want to use the algorithm without any custom modifications, the Installation of pre-built packages is usually a better choice.
Prerequisites¶
As a prerequisite, a supported version of Python, a suitable C++ compiler, as well as optional libraries for multi-threading and GPU support, must be available on the host system. The installation of these software components depends on the operation system at hand. In the following, we provide installation instructions for the supported platforms.
Tip
This project uses Meson as a build system for compiling C++ code. If available on the system, Meson automatically utilizes Ccache for caching previous compilations and detecting when the same compilation is being done again. Compared to the runtime without Ccache, where changes are only detected at the level of entire files, usage of this compiler cache can significantly speed up recompilation and therefore is strongly adviced.
Python |
Nowadays, most Linux distributions include a pre-installed version of Python 3. If this is not the case, instructions on how to install a recent Python version can be found in Python’s Beginners Guide. As noted in this guide, Python should be installed via the distribution’s package manager if possible. |
C++ compiler |
Most Linux distributions provide the GNU Compiler Collection (GCC), which includes a C++ compiler, as part of their software repositories. If this is the case, it can be installed via the distribution’s package manager. |
GoogleTest |
The GoogleTest framework must optionally be available in order to compile the project with Testing Support enabled. It should be possible to install it via the package manager of your Linux distribution. |
OpenMP |
OpenMP, which is optionally required for Multi-Threading Support, should be installable via your Linux distribution’s package manager. |
OpenCL |
If the project should be compiled with GPU Support, OpenCL must be available. On Linux, it should be installable via your distribution’s package manager. |
Python |
Recent versions of MacOS do not include Python by default. A suitable Python version can manually be downloaded from the project’s website. Alternatively, the package manager Homebrew can be used for installation via the command |
C++ compiler |
MacOS relies on the Clang compiler for building C++ code. It is part of the Xcode developer toolset. |
GoogleTest |
The GoogleTest framework must optionally be installed in order to compile the project with Testing Support enabled. It can easily be installed via Homebrew by runnig the command |
OpenMP |
If the project should be compiled with Multi-Threading Support enabled, the OpenMP library must be installed. We recommend to install it via Homebrew by running the command |
OpenCL |
The Xcode developer toolset should include OpenCL, which are needed for GPU Support. However, the OpenCL C++ headers must be installed manually. The easiest way to do so is via the Homebrew command |
Python |
Python releases for Windows are available at the project’s website, where you can download an installer. |
C++ compiler |
For the compilation of the project’s source code, the MSVC compiler must be used. It is included in Visual Studio. |
GoogleTest |
The GoogleTest framework must optionally be available on your system to compile the project with Testing Support enabled. It should already be included in recent versions of Visual Studio. |
OpenMP |
The Build Tools for Visual Studio also include the OpenMP library, which is utilized by the project for Multi-Threading Support. |
OpenCL |
If you intend to compile the project with GPU Support enabled, OpenCL must be installed manually. In order to do so, we recommend to install the package |
Additional build- or run-time dependencies are automatically installed when following the instructions below and must not be installed manually.
Tip
Instead of following the instructions below step by step, the following command, which automatically executes all necessary steps, can be used for simplicity.
./build
./build
build.bat
Whenever any C++, Cython or Python source files have been modified, the above command must be run again in order to rebuild modified files and install updated wheel packages into the virtual environment. If any compilation files already exist, this does only result in the affected parts of the code to be rebuilt.
Note
As shown in Project Structure, this project is organized in terms of several subprojects. By default, all of these subprojects are built when following the instructions below. However, the environment variable SUBPROJECTS may be used to restrict individual steps of the build process, such as the compilation of C++ and Cython code, the assemblage of Python packages, and the generation of apidocs, to a subset of the available subprojects. As shown below, multiple subprojects can be specified as a comma-separated list:
SUBPROJECTS=common,boosting ./build
SUBPROJECTS=common,boosting ./build
$env:SUBPROJECTS = "common,boosting"
build.bat
Creating a Virtual Environment¶
The build process is based on an virtual Python environment that allows to install build- and run-time dependencies in an isolated manner and independently from the host system. Once the build process was completed, the resulting Python packages are installed into the virtual environment. To create new virtual environment and install all necessarily run-time dependencies, the following command must be executed:
./build venv
./build venv
build.bat venv
All run-time dependencies (numpy, scipy, etc.) that are required for running the algorithms that are provided by the project should automatically be installed into the virtual environment when executing the above command. As a result, a subdirectory venv/ should have been created in the project’s root directory.
Compiling the C++ Code¶
Once a new virtual environment has successfully been created, the compilation of the C++ code can be started by executing the following command:
./build compile_cpp
./build compile_cpp
build.bat compile_cpp
The compilation is based on the build system Meson and uses Ninja as a backend. After the above command has terminated, a new directory cpp/build/ should have been created. It contains the shared libraries (“libmlrlcommon”, “libmlrlboosting” and possibly others) that provide the basic functionality of the project’s algorithms.
Compiling the Cython Code¶
Once the compilation of the C++ code has completed, the Cython code, which allows to access the corresponding shared libraries from within Python, can be compiled in the next step. Again, Meson and Ninja are used for compilation. It can be started via the following command:
./build compile_cython
./build compile_cython
build.bat compile_cython
As a result of executing the above command, the directory python/build should have been created. It contains Python extension modules for the respective target platform.
Note
Instead of performing the previous steps one after the other, the build target compile can be specfied instead of compile_cpp and compile_cython to build the C++ and Cython source files in a single step.
Installing Extension Modules¶
Similar to the previous step, the Python extension modules that have been built from the project’s Cython code must be copied into the Python source tree via the following command:
./build install_cython
./build install_cython
build.bat install_cython
As a result, the compilation files that can be found in the python/build/ directories should have been copied into the cython/ subdirectories of each Python module.
Note
Instead of executing the above commands one after the other, the build target install can be used instead of install_cpp and install_cython to copy both, the shared libraries and the extension modules, into the source tree.
Building Wheel Packages¶
Once the compilation files have been copied into the Python source tree, wheel packages can be built for the individual Python modules via the following command:
./build build_wheels
./build build_wheels
build.bat build_wheels
This should result in .whl files being created in a new dist/ subdirectory inside the directories that correspond to the individual Python modules (e.g., in the directory python/subprojects/common/dist/).
Installing the Wheel Packages¶
The wheel packages that have previously been created can finally be installed into the virtual environment via the following command:
./build install_wheels
./build install_wheels
build.bat install_wheels
After this final step has completed, the Python packages can be used from within the virtual environment once it has been activated. To ensure that the installation of the wheel packages was successful, check if a mlrl/ directory has been created in the lib/ directory of the virtual environment (depending on the Python version, it should be located at venv/lib/python3.9/site-packages/mlrl/ or similar). If this is the case, the algorithm can be used from within your own Python code. Alternatively, the command line API can be used to start an experiment (see Using the Command Line API).
Cleaning up Build Files¶
It is possible to delete the compilation files that result from an individual step of the build process mentioned above by using the command libe argument --clean or -c. This may be useful if you want to repeat a single or multiple steps of the build process from scratch in case anything went wrong. For example, to delete the C++ compilation files, the following command can be used:
./build --clean compile_cpp
./build --clean compile_cpp
build.bat --clean compile_cpp
If you want to delete all compilation files that have previously been created, including the virtual environment, you should use the following command, where no build target is specified:
./build --clean
./build --clean
build.bat --clean
Build Options¶
Certain functionalities of the project can be enabled or disabled at compile-time via so-called build options. They can be specified in the configuration file cpp/subprojects/common/meson.options or set via environment variables.
Testing Support¶
This project comes with unit tests for the C++ code it contains (see Testing the Code). They are based on the GoogleTest framework. When building the project on a system where this dependency is available, the testing code is compiled and linked against the shared libraries it is supposed to test. By default, the build option test_support is set to auto, i.e., the testing code is only compiled if GoogleTest is available and no error is raised otherwise. To enforce the compilation of the testing code, the build option can be set to enabled. Setting it to disabled prevents the code from being compiled even if GoogleTest is available. Alternatively, the desired value can be specified via the environment variable TEST_SUPPORT.
Multi-Threading Support¶
By default, the project is built with multi-threading support enabled. This requires OpenMP to be available on the host system. In order to compile the project without multi-threading support, e.g., because OpenMP is not available, the build option multi_threading_support can be set to disabled instead of enabled. Alternatively, the desired value can be specified via the environment variable MULTI_THREADING_SUPPORT.
When using the Using the Command Line API, the command boomer --version or boomer -v can be executed to check whether the program was built with multi-threading support enabled or not. It prints the build options used for compilation, as well as information about the CPU cores available on the system for multi-threading.
If you need to access this information programmatically in your own Python or C++ code, the following code snippets can be used (see Python API Reference and C++ API Reference):
from mlrl.common import get_num_cpu_cores, is_multi_threading_support_enabled
multi_threading_support_enabled: bool = is_multi_threading_support_enabled()
num_cpu_cores: int = get_num_cpu_cores()
#include "mlrl/common/info.hpp"
bool multiThreadingSupportEnabled = isMultiThreadingSupportEnabled();
uint32 numCpuCores = getNumCpuCores();
GPU Support¶
Warning
So far, GPU support is still at an early stage of development. No algorithm provided by this project makes use of it yet.
GPU support via OpenCL is enabled by default when building the project. However, it can be disabled at compile-time by setting the build option gpu_support to disabled instead of enabled. Alternatively, the desired value can be specified via the environment variable GPU_SUPPORT.
An easy way to check whether the program was built with GPU support enabled or not, is to run the boomer --version or boomer -v command that is provided by the Using the Command Line API. It prints the build options used for compiling the program, together with a list of supported GPUs available on your machine.
Alternatively, this information can be retrieved programmatically via the Python or C++ API as shown below (see Python API Reference and C++ API Reference):
from mlrl.common import get_gpu_devices, is_gpu_available, is_gpu_support_enabled
from typing import List
gpu_support_enabled: bool = is_gpu_support_enabled()
gpu_available: bool = is_gpu_available()
gpu_devices: List[str] = get_gpu_devices()
#include "mlrl/common/info.hpp"
bool gpuSupportEnabled = isGpuSupportEnabled();
bool gpuAvailable = isGpuAvailable();
std::vector<std::string> gpuDevices = getGpuDevices();