mlrl.testbed.experiments package¶

Author Michael Rapp (michael.rapp.ml@gmail.com)

Provides classes that allow running experiments.

class mlrl.testbed.experiments.ClassificationProblem¶

Bases: ProblemDomain, ABC

Represents a classification problem.

NAME = 'classification'¶
property problem_name: str¶

See mlrl.testbed.experiments.problem_domain.ProblemDomain.learner_name()

class mlrl.testbed.experiments.Experiment(args: Namespace, initial_state: ExperimentState, dataset_splitter: DatasetSplitter, training_procedure: TrainingProcedure, prediction_procedure: PredictionProcedure)¶

Bases: ABC

An abstract base class for all experiments that train and evaluate a machine learning model.

class Builder(initial_state: ExperimentState, dataset_splitter: DatasetSplitter)¶

Bases: ABC

An abstract base class for all classes that allow to configure and create instances of an experiment.

Factory¶

alias of Callable[[Namespace], Experiment.Builder]

add_before_start_output_writers(*output_writers: OutputWriter) → Builder¶

Adds one or several output writers that should be invoked before an experiment is started.

Parameters:

output_writers – The output writers to be added

Returns:

The builder itself

add_input_readers(*input_readers: InputReader) → Builder¶

Adds one or several input readers that should be invoked when the experiment is started.

Parameters:

input_readers – The input readers to be added

Returns:

The builder itself

add_listeners(*listeners: ExperimentListener) → Builder¶

Adds one or several listeners that should be informed about certain events during the experiment.

Parameters:

listeners – The listeners that should be added

Returns:

The builder itself

add_post_training_output_writers(*output_writers: OutputWriter) → Builder¶

Adds one or several output writers that should be invoked after a machine learning model has been trained.

Parameters:

output_writers – The output writers to be added

Returns:

The builder itself

add_pre_training_output_writers(*output_writers: OutputWriter) → Builder¶

Adds one or several output writers that should be invoked before a machine learning model is trained.

Parameters:

output_writers – The output writers to be added

Returns:

The builder itself

add_prediction_output_writers(*output_writers: OutputWriter) → Builder¶

Adds one or several output writers that should be invoked after predictions have been obtained from a machine learning model.

Parameters:

output_writers – The output writers to be added

Returns:

The builder itself

build(args: Namespace) → Experiment¶

Creates and returns a new experiment according to the specified configuration.

Parameters:

args – The command line arguments specified by the user

Returns:

The experiment that has been created

property has_output_file_writers: bool¶

True, if any output writers that write to output files have been added to the builder, False otherwise.

property output_writers: chain[OutputWriter]¶

A generator that provides access to all output writers that have been added to the builder.

run(args: Namespace)¶

Creates and runs a new experiment according to the specified configuration.

Parameters:

args – The command line arguments specified by the user

set_missing_input_policy(missing_input_policy: MissingInputPolicy) → Builder¶

Sets the policy to be used if an error occurs while reading input data.

Parameters:

missing_input_policy – The policy to be set

Returns:

The builder itself

set_output_error_policy(output_error_policy: OutputErrorPolicy) → Builder¶

Sets the policy to be used if an error occurs while writing experimental results.

Parameters:

output_error_policy – The policy to be set

Returns:

The builder itself

set_predict_for_test_dataset(predict_for_test_dataset: bool) → Builder¶

Sets whether predictions should be obtained for the test dataset, if available, or not.

Parameters:

predict_for_test_dataset – True, if predictions should be obtained for the test dataset, if available, False otherwise

Returns:

The builder itself

set_predict_for_training_dataset(predict_for_training_dataset: bool) → Builder¶

Sets whether predictions should be obtained for the training dataset or not.

Parameters:

predict_for_training_dataset – True, if predictions should be obtained for the training dataset, False otherwise

Returns:

The builder itself

class InputReaderListener(experiment: Experiment, args: Namespace)¶

Bases: ExperimentListener

Updates the state of an experiment by invoking the input readers that have been added to an experiment.

on_start(state: ExperimentState) → ExperimentState¶

May be overridden by subclasses in order to be notified when an experiment has been started on a specific dataset. May be called multiple times if several datasets are used.

Parameters:

state – The current state of the experiment

Returns:

An update of the given state

class OutputWriterListener(experiment: Experiment)¶

Bases: ExperimentListener

Passes the state of an experiment to output writers that have been added to an experiment.

after_prediction(state: ExperimentState)¶

May be overridden by subclasses in order to be notified after predictions for a dataset have been obtained from a machine learning model. May be called multiple times if predictions are obtained for several datasets.

Parameters:

state – The current state of the experiment

Returns:

An update of the given state

after_training(state: ExperimentState) → ExperimentState¶

May be overridden by subclasses in order to be notified after a machine learning model has been trained.

Parameters:

state – The current state of the experiment

Returns:

An update of the given state

before_start(state: ExperimentState)¶

May be overridden by subclasses in order to be notified just before the experiment starts.

Parameters:

state – The current state of the experiment

Returns:

An update of the given state

before_training(state: ExperimentState) → ExperimentState¶

May be overridden by subclasses in order to be notified before a machine learning model is trained.

Parameters:

state – The current state of the experiment

Returns:

An update of the given state

class PredictionProcedure¶

Bases: ABC

An abstract base class for all classes that allow to obtain predictions for given query examples from a previously trained learner.

abstractmethod predict(state: ExperimentState) → Generator[PredictionState, None, None]¶

Obtains predictions for given query examples from a previously trained learner.

Parameters:

state – The current state of the experiment

Returns:

The PredictionState that stores the result of the prediction process

class TrainingProcedure¶

Bases: ABC

An abstract base class for all classes that allow to fit a learner to a training dataset.

abstractmethod train(learner: Any | None, parameters: dict[str, Any], dataset: Any) → TrainingState¶

Fits a learner to a training dataset.

Parameters:

  • learner – An existing learner or None, if a new learner must be trained from scratch

  • parameters – The algorithmic parameters to be used

  • dataset – The training dataset

Returns:

A TrainingState that stores the result of the training process

class mlrl.testbed.experiments.ProblemDomain¶

Bases: ABC

An abstract base class for all classes that represent a specific problem domain.

abstract property learner_name: str¶

The name of the machine learning algorithm used to tackle the problem domain.

abstract property problem_name: str¶

The name of the problem domain.

class mlrl.testbed.experiments.RegressionProblem¶

Bases: ProblemDomain, ABC

Represents a regression problem.

NAME = 'regression'¶
property problem_name: str¶

See mlrl.testbed.experiments.problem_domain.ProblemDomain.learner_name()

Subpackages¶

Submodules¶