This software package provides mlrl-testbed - a command line utility for running machine learning experiments. It implements a straightforward, easily configurable, and extensible workflow for conducting experiments, including steps such as (but not restricted to) the following:

loading a dataset
splitting it into training and test sets
training one or several models
evaluating the models’ performance
saving experimental results to output files

MLRL-Testbed¶

On its own, this package is not very powerful. It is intended as a basis for other packages that build functionality upon it. In fact, it does not make any assumptions about the problem domain or type of machine learning algorithm that should be used in an experiment. Instead, implementations of domain- or algorithm-specific functionality are provided by the extensions discussed below.

Tabular Machine Learning¶

The package mlrl-testbed-sklearn adds support for tabular machine learning problems by making use of the scikit-learn framework. It can easily be installed via the following command (and will pull mlrl-testbed as a dependency):

pip install mlrl-testbed-sklearn

💡 Example¶

By writing just a small amount of code, any scikit-learn compatible estimator can be integrated with MLRL-Testbed and used in experiments. For example, the following code integrates scikit-learn’s RandomForestClassifier:

from argparse import Namespace
from mlrl.testbed_sklearn.runnables import SkLearnRunnable
from mlrl.util.cli import Argument, IntArgument
from sklearn.ensemble import RandomForestClassifier
from sklearn.base import ClassifierMixin, RegressorMixin
from typing import Optional, Set


class Runnable(SkLearnRunnable):

    N_ESTIMATORS = IntArgument(
        '--n-estimators',
        description='The number of trees in the forest',
        default=100,
    )

    def get_algorithmic_arguments(self, known_args: Namespace) -> Set[Argument]:
        return { self.N_ESTIMATORS }

    def create_classifier(self, args: Namespace) -> Optional[ClassifierMixin]:
        return RandomForestClassifier()

    def create_regressor(self, args: Namespace) -> Optional[RegressorMixin]:
        return None  # Not needed in this case

The previously integrated algorithm can then be used in experiments controlled via a command line API. Assuming that the source code shown above is saved to a file named custom_runnable.py in the working directory, we are now capable of fitting a RandomForestClassifier to a dataset by using the command below.

mlrl-testbed custom_runnable.py \
    --data-dir path/to/datasets/ \
    --dataset dataset-name \
    --n-estimators 50

The above command does not only train a model, but also evaluates it according to common measures and prints the evaluation results. It does also demonstrate how algorithmic parameters can be controlled via command line arguments.

🏁 Advantages¶

Making use of MLRL-Testbed does not only help with the burdens of training and evaluating machine learning models, it can also help making your own methods and algorithms more accessible to users. This is demonstrated by the rule learning algorithms mlrl-boomer and mlrl-seco that can easily be run via the command line API described above and even extend it with rule-specific functionalities.

🔧 Functionalities¶

The package mlrl-testbed-sklearn provides a command line API that allows configuring and running machine learning algorithms. It allows to apply machine learning algorithms to different datasets and can evaluate their predictive performance in terms of commonly used measures. In detail, it supports the following functionalities:

Single- and multi-output datasets in the Mulan and Meka format are supported (with the help of the package mlrl-testbed-arff).
Datasets can automatically be split into training and test data, including the possibility to use cross validation. Alternatively, predefined splits can be provided as separate files.
One-hot-encoding can be applied to nominal or binary features.
Binary predictions, scores, or probability estimates can be obtained from machine learning algorithms, if supported. Evaluation measures that are suited for the respective type of predictions are picked automatically.

Furthermore, the command line API provides many options for controlling the experimental results to be gathered during an experiment. Depending on the configuration, the following experimental results can be saved to output files or printed on the console:

Evaluation scores according to commonly used measures
Characteristics, i.e., statistical properties, of datasets
Predictions and their characteristics
Unique label vectors contained in a classification dataset

If the following are written to output files, they can be loaded and reused in future experiments:

The machine learning models that have been learned
Algorithmic parameters used for training

📚 Documentation¶

This documentation provides information about the following topics:

If you are a developer, you might also be interested in the API reference.