mlrl.testbed.experiments.experiment module

Author: Michael Rapp (michael.rapp.ml@gmail.com)

Provides classes for implementing experiments.

class mlrl.testbed.experiments.experiment.Experiment(problem_domain: ProblemDomain, dataset_splitter: DatasetSplitter)

Bases: ABC

An abstract base class for all experiments that train and evaluate a machine learning model.

class Builder(problem_domain: ProblemDomain, dataset_splitter: DatasetSplitter)

Bases: ABC

An abstract base class for all classes that allow to configure and create instances of an experiment.

Factory

alias of Callable[[Namespace], Experiment.Builder]

add_input_readers(*input_readers: InputReader) → Builder

Adds one or several input readers that should be invoked when the experiment is started.

Parameters:

input_readers – The input readers to be added

Returns:

The builder itself

add_listeners(*listeners: Listener) → Builder

Adds one or several listeners that should be informed about certain events during the experiment.

Parameters:

listeners – The listeners that should be added

Returns:

The builder itself

add_post_training_output_writers(*output_writers: OutputWriter) → Builder

Adds one or several output writers that should be invoked after a machine learning model has been trained.

Parameters:

output_writers – The output writers to be added

Returns:

The builder itself

add_pre_training_output_writers(*output_writers: OutputWriter) → Builder

Adds one or several output writers that should be invoked before a machine learning model is trained.

Parameters:

output_writers – The output writers to be added

Returns:

The builder itself

add_prediction_output_writers(*output_writers: OutputWriter) → Builder

Adds one or several output writers that should be invoked after predictions have been obtained from a machine learning model.

Parameters:

output_writers – The output writers to be added

Returns:

The builder itself

build() → Experiment

Creates and returns a new experiment according to the specified configuration.

Returns:

The experiment that has been created

run()

Creates and runs a new experiment according to the specified configuration.

set_exit_on_error(exit_on_error: bool) → Builder

Sets whether the program should exit if an error occurs while writing experimental results.

Parameters:

exit_on_error – True, if the program should be aborted if an error occurs, False otherwise

Returns:

The builder itself

set_predict_for_test_dataset(predict_for_test_dataset: bool) → Builder

Sets whether predictions should be obtained for the test dataset, if available, or not.

Parameters:

predict_for_test_dataset – True, if predictions should be obtained for the test dataset, if available, False otherwise

Returns:

The builder itself

set_predict_for_training_dataset(predict_for_training_dataset: bool) → Builder

Sets whether predictions should be obtained for the training dataset or not.

Parameters:

predict_for_training_dataset – True, if predictions should be obtained for the training dataset, False otherwise

Returns:

The builder itself

class InputReaderListener

Bases: Listener

Updates the state of an experiment by invoking the input readers that have been added to an experiment.

on_start(experiment: Experiment, state: ExperimentState) → ExperimentState

May be overridden by subclasses in order to be notified when an experiment has been started on a specific dataset. May be called multiple times if several datasets are used.

Parameters:

  • experiment – The experiment

  • state – The initial state of the experiment

Returns:

An update of the given state

class Listener

Bases: ABC

An abstract base class for all listeners that may be informed about certain event during an experiment.

after_prediction(experiment: Experiment, state: ExperimentState)

May be overridden by subclasses in order to be notified after predictions for a dataset have been obtained from a machine learning model. May be called multiple times if predictions are obtained for several datasets.

Parameters:

  • experiment – The experiment

  • state – The current state of the experiment

after_training(experiment: Experiment, state: ExperimentState) → ExperimentState

May be overridden by subclasses in order to be notified after a machine learning model has been trained.

Parameters:

  • experiment – The experiment

  • state – The current state of the experiment

Returns:

An update of the given state

before_start(experiment: Experiment)

May be overridden by subclasses in order to be notified just before the experiment starts.

Parameters:

experiment – The experiment

before_training(experiment: Experiment, state: ExperimentState) → ExperimentState

May be overridden by subclasses in order to be notified before a machine learning model is trained.

Parameters:

  • experiment – The experiment

  • state – The current state of the experiment

Returns:

An update of the given state

on_start(experiment: Experiment, state: ExperimentState) → ExperimentState

May be overridden by subclasses in order to be notified when an experiment has been started on a specific dataset. May be called multiple times if several datasets are used.

Parameters:

  • experiment – The experiment

  • state – The initial state of the experiment

Returns:

An update of the given state

class OutputWriterListener

Bases: Listener

Passes the state of an experiment to output writers that have been added to an experiment.

after_prediction(experiment: Experiment, state: ExperimentState)

May be overridden by subclasses in order to be notified after predictions for a dataset have been obtained from a machine learning model. May be called multiple times if predictions are obtained for several datasets.

Parameters:

  • experiment – The experiment

  • state – The current state of the experiment

after_training(experiment: Experiment, state: ExperimentState) → ExperimentState

May be overridden by subclasses in order to be notified after a machine learning model has been trained.

Parameters:

  • experiment – The experiment

  • state – The current state of the experiment

Returns:

An update of the given state

before_training(experiment: Experiment, state: ExperimentState) → ExperimentState

May be overridden by subclasses in order to be notified before a machine learning model is trained.

Parameters:

  • experiment – The experiment

  • state – The current state of the experiment

Returns:

An update of the given state

run(predict_for_training_dataset: bool, predict_for_test_dataset: bool)

Runs the experiment.

Parameters:

  • predict_for_training_dataset – True, if predictions should be obtained for the training dataset, False otherwise

  • predict_for_test_dataset – True, if predictions should be obtained for the test dataset, if available, False otherwise