mlrl.testbed.modes.mode_batch module

Author: Michael Rapp (michael.rapp.ml@gmail.com)

Provides classes that implement a mode of operation for performing multiple experiments.

class mlrl.testbed.modes.mode_batch.BatchMode(config_file_factory: Callable[[str], ConfigFile] | None = None)

Bases: Mode

An abstract base class for all modes of operation that perform multiple experiments.

CONFIG_FILE = <mlrl.util.cli.StringArgument object>
class ConfigFile(file_path: str, schema_file_path: str)

Bases: ABC

A YAML configuration file that configures the batch of experiments to be run.

Factory

alias of Callable[[str], BatchMode.ConfigFile]

class Parameter(name: str, values: List[ParameterValue])

Bases: object

A parameter defined in the configuration file.

Attributes:

name: The name of the parameter values: One or several values that should be set for the parameter

name: str
values: List[ParameterValue]
class ParameterValue(value: str, additional_arguments: ~typing.List[str] = <factory>)

Bases: object

A value that should be set for a parameter.

Attributes:

value: The value to be set additional_arguments: Additional arguments associated with the parameter

additional_arguments: List[str]
value: str
abstract property dataset_args: List[ArgumentList]

A list that contains the command line arguments corresponding to the datasets to be used in the different experiments defined in the configuration file.

property parameter_args: List[ArgumentList]

A list that contains the command line arguments corresponding to the algorithmic parameters to be used in the different experiments defined in the configuration file.

property parameters: List[Parameter]

A list that contains all parameters defined in the configuration file.

LIST_COMMANDS = <mlrl.util.cli.FlagArgument object>
class LogRunner

Bases: Runner

Allows to log the commands of experiments, instead of executing them.

run_batch(args: Namespace, batch: List[Command], recipe: Recipe)

Must be implemented by subclasses in order to run all commands in a given batch.

Parameters:

  • args – The command line arguments specified by the user

  • batch – The batch of experiments to be run

  • recipe – A Recipe that provides access to the ingredients that are needed for setting up experiments

RUNNER = <mlrl.util.cli.SetArgument object>
class Runner(name: str)

Bases: ABC

” An abstract base class for all classes that allow to run experiments.

abstractmethod run_batch(args: Namespace, batch: List[Command], recipe: Recipe)

Must be implemented by subclasses in order to run all commands in a given batch.

Parameters:

  • args – The command line arguments specified by the user

  • batch – The batch of experiments to be run

  • recipe – A Recipe that provides access to the ingredients that are needed for setting up experiments

SEPARATE_FOLDS = <mlrl.util.cli.BoolArgument object>
class SequentialRunner

Bases: Runner

Allows to run experiments sequentially.

run_batch(args: Namespace, batch: List[Command], recipe: Recipe)

Must be implemented by subclasses in order to run all commands in a given batch.

Parameters:

  • args – The command line arguments specified by the user

  • batch – The batch of experiments to be run

  • recipe – A Recipe that provides access to the ingredients that are needed for setting up experiments

add_runner(runner: Runner) → BatchMode

Adds a runner that may be used in batch mode for running individual experiments in a batch.

Parameters:

runner – The runner to be added

Returns:

The batch mode itself

configure_arguments(cli: CommandLineInterface)

Must be implemented by subclasses in order to configure the command line interface according to the mode of operation.

Parameters:

cli – The command line interface to be configured

run_experiment(args: Namespace, recipe: Recipe)

Must be implemented by subclasses in order to run an experiment according to the command line arguments specified by the user.

Parameters:

  • args – The command line arguments specified by the user

  • recipe – A Recipe that provides access to the ingredients that are needed for setting up experiments

property runners: List[Runner]

The runners that may be used for running individual experiments in a batch.