mlrl.testbed.experiments.experiment module

Author: Michael Rapp (michael.rapp.ml@gmail.com)

Provides classes for implementing experiments.

class mlrl.testbed.experiments.experiment.DefaultProcedure(predict_for_training_dataset: bool, predict_for_test_dataset: bool)

Bases: ExperimentalProcedure

Implements the default procedure for conducting experiments.

EXTRA_START_TIME = 'start_time'

class mlrl.testbed.experiments.experiment.Experiment(args: Namespace, initial_state: ExperimentState, dataset_splitter: DatasetSplitter, training_procedure: TrainingProcedure, prediction_procedure: PredictionProcedure)

Bases: ABC

An abstract base class for all experiments that train and evaluate a machine learning model.

class Builder(initial_state: ExperimentState, dataset_splitter: DatasetSplitter)

Bases: ABC

An abstract base class for all classes that allow to configure and create instances of an experiment.

Factory

alias of Callable[[Namespace], Experiment.Builder]

add_before_start_output_writers(*output_writers: OutputWriter) → Builder

Adds one or several output writers that should be invoked before an experiment is started.

Parameters:

output_writers – The output writers to be added

Returns:

The builder itself

add_input_readers(*input_readers: InputReader) → Builder

Adds one or several input readers that should be invoked when the experiment is started.

Parameters:

input_readers – The input readers to be added

Returns:

The builder itself

add_listeners(*listeners: ExperimentListener) → Builder

Adds one or several listeners that should be informed about certain events during the experiment.

Parameters:

listeners – The listeners that should be added

Returns:

The builder itself

add_post_training_output_writers(*output_writers: OutputWriter) → Builder

Adds one or several output writers that should be invoked after a machine learning model has been trained.

Parameters:

output_writers – The output writers to be added

Returns:

The builder itself

add_pre_training_output_writers(*output_writers: OutputWriter) → Builder

Adds one or several output writers that should be invoked before a machine learning model is trained.

Parameters:

output_writers – The output writers to be added

Returns:

The builder itself

add_prediction_output_writers(*output_writers: OutputWriter) → Builder

Adds one or several output writers that should be invoked after predictions have been obtained from a machine learning model.

Parameters:

output_writers – The output writers to be added

Returns:

The builder itself

before_start_output_writers: set[OutputWriter]

build(args: Namespace) → Experiment

Creates and returns a new experiment according to the specified configuration.

Parameters:

args – The command line arguments specified by the user

Returns:

The experiment that has been created

property has_output_file_writers: bool

True, if any output writers that write to output files have been added to the builder, False otherwise.

input_readers: list[InputReader]

listeners: list[ExperimentListener]

property output_writers: chain[OutputWriter]

A generator that provides access to all output writers that have been added to the builder.

post_training_output_writers: set[OutputWriter]

pre_training_output_writers: set[OutputWriter]

prediction_output_writers: set[OutputWriter]

run(args: Namespace)

Creates and runs a new experiment according to the specified configuration.

Parameters:

args – The command line arguments specified by the user

set_missing_input_policy(missing_input_policy: MissingInputPolicy) → Builder

Sets the policy to be used if an error occurs while reading input data.

Parameters:

missing_input_policy – The policy to be set

Returns:

The builder itself

set_output_error_policy(output_error_policy: OutputErrorPolicy) → Builder

Sets the policy to be used if an error occurs while writing experimental results.

Parameters:

output_error_policy – The policy to be set

Returns:

The builder itself

set_predict_for_test_dataset(predict_for_test_dataset: bool) → Builder

Sets whether predictions should be obtained for the test dataset, if available, or not.

Parameters:

predict_for_test_dataset – True, if predictions should be obtained for the test dataset, if available, False otherwise

Returns:

The builder itself

set_predict_for_training_dataset(predict_for_training_dataset: bool) → Builder

Sets whether predictions should be obtained for the training dataset or not.

Parameters:

predict_for_training_dataset – True, if predictions should be obtained for the training dataset, False otherwise

Returns:

The builder itself

class InputReaderListener(experiment: Experiment, args: Namespace)

Bases: ExperimentListener

Updates the state of an experiment by invoking the input readers that have been added to an experiment.

on_start(state: ExperimentState) → ExperimentState

May be overridden by subclasses in order to be notified when an experiment has been started on a specific dataset. May be called multiple times if several datasets are used.

Parameters:

state – The current state of the experiment

Returns:

An update of the given state

class OutputWriterListener(experiment: Experiment)

Bases: ExperimentListener

Passes the state of an experiment to output writers that have been added to an experiment.

after_prediction(state: ExperimentState)

May be overridden by subclasses in order to be notified after predictions for a dataset have been obtained from a machine learning model. May be called multiple times if predictions are obtained for several datasets.

Parameters:

state – The current state of the experiment

Returns:

An update of the given state

after_training(state: ExperimentState) → ExperimentState

May be overridden by subclasses in order to be notified after a machine learning model has been trained.

Parameters:

state – The current state of the experiment

Returns:

An update of the given state

before_start(state: ExperimentState)

May be overridden by subclasses in order to be notified just before the experiment starts.

Parameters:

state – The current state of the experiment

Returns:

An update of the given state

before_training(state: ExperimentState) → ExperimentState

May be overridden by subclasses in order to be notified before a machine learning model is trained.

Parameters:

state – The current state of the experiment

Returns:

An update of the given state

class PredictionProcedure

Bases: ABC

An abstract base class for all classes that allow to obtain predictions for given query examples from a previously trained learner.

abstractmethod predict(state: ExperimentState) → Generator[PredictionState, None, None]

Obtains predictions for given query examples from a previously trained learner.

Parameters:

state – The current state of the experiment

Returns:

The PredictionState that stores the result of the prediction process

class TrainingProcedure

Bases: ABC

An abstract base class for all classes that allow to fit a learner to a training dataset.

abstractmethod train(learner: Any | None, parameters: dict[str, Any], dataset: Any) → TrainingState

Fits a learner to a training dataset.

Parameters:

• learner – An existing learner or None, if a new learner must be trained from scratch

• parameters – The algorithmic parameters to be used

• dataset – The training dataset

Returns:

A TrainingState that stores the result of the training process

before_start_output_writers: list[OutputWriter]

input_readers: list[InputReader]

listeners: list[ExperimentListener]

post_training_output_writers: list[OutputWriter]

pre_training_output_writers: list[OutputWriter]

prediction_output_writers: list[OutputWriter]

class mlrl.testbed.experiments.experiment.ExperimentListener

Bases: ABC

An abstract base class for all listeners that may be informed about certain event during an experiment.

after_prediction(state: ExperimentState) → ExperimentState

May be overridden by subclasses in order to be notified after predictions for a dataset have been obtained from a machine learning model. May be called multiple times if predictions are obtained for several datasets.

Parameters:

state – The current state of the experiment

Returns:

An update of the given state

after_training(state: ExperimentState) → ExperimentState

May be overridden by subclasses in order to be notified after a machine learning model has been trained.

Parameters:

state – The current state of the experiment

Returns:

An update of the given state

before_start(state: ExperimentState) → ExperimentState

May be overridden by subclasses in order to be notified just before the experiment starts.

Parameters:

state – The current state of the experiment

Returns:

An update of the given state

before_training(state: ExperimentState) → ExperimentState

May be overridden by subclasses in order to be notified before a machine learning model is trained.

Parameters:

state – The current state of the experiment

Returns:

An update of the given state

on_start(state: ExperimentState) → ExperimentState

May be overridden by subclasses in order to be notified when an experiment has been started on a specific dataset. May be called multiple times if several datasets are used.

Parameters:

state – The current state of the experiment

Returns:

An update of the given state

class mlrl.testbed.experiments.experiment.ExperimentalProcedure

Bases: ABC

An abstract base class for all classes that implement procedures for conducting experiments.

conduct_experiment(experiment: Experiment) → ExperimentState

Conducts a given experiment.

Parameters:

experiment – The experiment to be conducted

Returns:

The final state of the experiment