mlrl.testbed.modes package

Author Michael Rapp (michael.rapp.ml@gmail.com)

Provides classes for implementing different modes of operation.

class mlrl.testbed.modes.BatchMode(config_file_factory: Callable[[str], ConfigFile] | None = None)

Bases: Mode

An abstract base class for all modes of operation that perform multiple experiments.

ARGUMENT_RUNNER = '--runner'
CONFIG_FILE = <mlrl.util.cli.PathArgument object>
class ConfigFile(file_path: Path, schema_file_path: Path)

Bases: ABC

A YAML configuration file that configures the batch of experiments to be run.

class DefaultArgument(name: str, value: str)

Bases: object

A default parameter defined in the configuration file.

Attributes:

name: The name of the parameter value: The value of the parameter

name: str
value: str
Factory

alias of Callable[[str], BatchMode.ConfigFile]

class Parameter(name: str, values: list[ParameterValue])

Bases: object

A parameter defined in the configuration file.

Attributes:

name: The name of the parameter values: One or several values that should be set for the parameter

name: str
values: list[ParameterValue]
class ParameterValue(value: str, additional_arguments: list[str] = <factory>)

Bases: object

A value that should be set for a parameter.

Attributes:

value: The value to be set additional_arguments: Additional arguments associated with the parameter

additional_arguments: list[str]
value: str
abstract property dataset_args: list[ArgumentList]

A list that contains the command line arguments corresponding to the datasets to be used in the different experiments defined in the configuration file.

property default_arguments: list[DefaultArgument]

A list that contains all default arguments defined in the configuration file.

property parameter_args: list[ArgumentList]

A list that contains the command line arguments corresponding to the algorithmic parameters to be used in the different experiments defined in the configuration file.

property parameters: list[Parameter]

A list that contains all parameters defined in the configuration file.

LIST_COMMANDS = <mlrl.util.cli.FlagArgument object>
class LogRunner

Bases: Runner

Allows to log the commands of experiments, instead of executing them.

run_batch(args: Namespace, batch: list[Command], recipe: Recipe)

Must be implemented by subclasses in order to run all commands in a given batch.

Parameters:

  • args – The command line arguments specified by the user

  • batch – The batch of experiments to be run

  • recipe – A Recipe that provides access to the ingredients that are needed for setting up experiments

class Runner(name: str)

Bases: ABC

An abstract base class for all classes that allow to run experiments.

abstractmethod run_batch(args: Namespace, batch: list[Command], recipe: Recipe)

Must be implemented by subclasses in order to run all commands in a given batch.

Parameters:

  • args – The command line arguments specified by the user

  • batch – The batch of experiments to be run

  • recipe – A Recipe that provides access to the ingredients that are needed for setting up experiments

SEPARATE_FOLDS = <mlrl.util.cli.BoolArgument object>
class SequentialRunner

Bases: Runner

Allows to run experiments sequentially.

run_batch(args: Namespace, batch: list[Command], recipe: Recipe)

Must be implemented by subclasses in order to run all commands in a given batch.

Parameters:

  • args – The command line arguments specified by the user

  • batch – The batch of experiments to be run

  • recipe – A Recipe that provides access to the ingredients that are needed for setting up experiments

add_runner(runner: Runner) → BatchMode

Adds a runner that may be used in batch mode for running individual experiments in a batch.

Parameters:

runner – The runner to be added

Returns:

The batch mode itself

configure_algorithmic_arguments(cli: CommandLineInterface, algorithmic_arguments: list[Argument])

Must be implemented by subclasses in order to configure the command line interface according to the mode of operation.

Parameters:

  • cli – The command line interface to be configured

  • algorithmic_arguments – The arguments that should be added to the command line interface for configuring the algorithm’s hyperparameters

configure_control_arguments(cli: CommandLineInterface, control_arguments: list[Argument])

Must be implemented by subclasses in order to configure the command line interface according to the mode of operation.

Parameters:

  • cli – The command line interface to be configured

  • control_arguments – The arguments that should be added to the command line interface for controlling mlrl-testbed’s behavior

run_experiment(control_arguments: set[Argument], algorithmic_arguments: set[Argument], args: Namespace, recipe: Recipe)

Must be implemented by subclasses in order to run an experiment according to the command line arguments specified by the user.

Parameters:

  • control_arguments – The arguments that should be added to the command line interface for controlling mlrl-testbed’s behavior

  • algorithmic_arguments – The arguments that should be added to the command line interface for configuring the algorithm’s hyperparameters

  • args – The command line arguments specified by the user

  • recipe – A Recipe that provides access to the ingredients that are needed for setting up experiments

property runners: list[Runner]

The runners that may be used for running individual experiments in a batch.

to_enum() → ExperimentMode

Returns the ExperimentMode that corresponds to this mode of operation.

Returns:

An ExperimentMode

class mlrl.testbed.modes.Mode

Bases: ABC

An abstract base class for all modes of operation.

MODE = <mlrl.util.cli.SetArgument object>
abstractmethod configure_algorithmic_arguments(cli: CommandLineInterface, algorithmic_arguments: list[Argument])

Must be implemented by subclasses in order to configure the command line interface according to the mode of operation.

Parameters:

  • cli – The command line interface to be configured

  • algorithmic_arguments – The arguments that should be added to the command line interface for configuring the algorithm’s hyperparameters

abstractmethod configure_control_arguments(cli: CommandLineInterface, control_arguments: list[Argument])

Must be implemented by subclasses in order to configure the command line interface according to the mode of operation.

Parameters:

  • cli – The command line interface to be configured

  • control_arguments – The arguments that should be added to the command line interface for controlling mlrl-testbed’s behavior

abstractmethod run_experiment(control_arguments: set[Argument], algorithmic_arguments: set[Argument], args: Namespace, recipe: Recipe)

Must be implemented by subclasses in order to run an experiment according to the command line arguments specified by the user.

Parameters:

  • control_arguments – The arguments that should be added to the command line interface for controlling mlrl-testbed’s behavior

  • algorithmic_arguments – The arguments that should be added to the command line interface for configuring the algorithm’s hyperparameters

  • args – The command line arguments specified by the user

  • recipe – A Recipe that provides access to the ingredients that are needed for setting up experiments

abstractmethod to_enum() → ExperimentMode

Returns the ExperimentMode that corresponds to this mode of operation.

Returns:

An ExperimentMode

class mlrl.testbed.modes.ReadMode

Bases: InputMode

An abstract base class for all modes of operation that read experimental results.

class AggregateEvaluationProcedure(evaluation_by_dataset_type: dict[DatasetType, dict[str, Table]])

Bases: ExperimentalProcedure

The procedure that is used to write evaluation results that have been aggregated across several experiments to one or several sinks.

to_enum() → ExperimentMode

Returns the ExperimentMode that corresponds to this mode of operation.

Returns:

An ExperimentMode

class mlrl.testbed.modes.RunMode

Bases: InputMode

An abstract base class for all modes of operation for re-running experiments based on their meta-data.

to_enum() → ExperimentMode

Returns the ExperimentMode that corresponds to this mode of operation.

Returns:

An ExperimentMode

class mlrl.testbed.modes.SingleMode

Bases: Mode

A mode of operation that performs a single experiment.

configure_algorithmic_arguments(cli: CommandLineInterface, algorithmic_arguments: list[Argument])

Must be implemented by subclasses in order to configure the command line interface according to the mode of operation.

Parameters:

  • cli – The command line interface to be configured

  • algorithmic_arguments – The arguments that should be added to the command line interface for configuring the algorithm’s hyperparameters

configure_control_arguments(cli: CommandLineInterface, control_arguments: list[Argument])

Must be implemented by subclasses in order to configure the command line interface according to the mode of operation.

Parameters:

  • cli – The command line interface to be configured

  • control_arguments – The arguments that should be added to the command line interface for controlling mlrl-testbed’s behavior

run_experiment(control_arguments: set[Argument], algorithmic_arguments: set[Argument], args: Namespace, recipe: Recipe)

Must be implemented by subclasses in order to run an experiment according to the command line arguments specified by the user.

Parameters:

  • control_arguments – The arguments that should be added to the command line interface for controlling mlrl-testbed’s behavior

  • algorithmic_arguments – The arguments that should be added to the command line interface for configuring the algorithm’s hyperparameters

  • args – The command line arguments specified by the user

  • recipe – A Recipe that provides access to the ingredients that are needed for setting up experiments

to_enum() → ExperimentMode

Returns the ExperimentMode that corresponds to this mode of operation.

Returns:

An ExperimentMode

Submodules