mlrl.testbed.experiments module

Author: Michael Rapp (michael.rapp.ml@gmail.com)

Provides classes for performing experiments.

class mlrl.testbed.experiments.Evaluation(prediction_type: PredictionType, output_writers: List[OutputWriter])

Bases: ABC

An abstract base class for all classes that allow to evaluate predictions that are obtained from a previously trained model.

abstract predict_and_evaluate(problem_type: ProblemType, meta_data: MetaData, data_split: DataSplit, data_type: DataType, train_time: float, learner, x, y, **kwargs)

Must be implemented by subclasses in order to obtain and evaluate predictions for given query examples from a previously trained model.

Parameters:

  • problem_type – The type of the machine learning problem

  • meta_data – The meta-data of the data set

  • data_split – The split of the available data, the predictions and ground truth correspond to

  • data_type – Specifies whether the predictions and ground truth correspond to the training or test data

  • train_time – The time needed to train the model

  • learner – The learner, the predictions should be obtained from

  • x – A numpy.ndarray, scipy.sparse.spmatrix or scipy.sparse.sparray, shape (num_examples, num_features), that stores the feature values of the query examples

  • y – A numpy.ndarray, scipy.sparse.spmatrix or scipy.sparse.sparray, shape (num_examples, num_outputs), that stores the ground truth of the query examples

  • kwargs – Optional keyword arguments to be passed to the model when obtaining predictions

class mlrl.testbed.experiments.Experiment(problem_type: ProblemType, base_learner: BaseEstimator, learner_name: str, data_splitter: DataSplitter, pre_training_output_writers: List[OutputWriter], post_training_output_writers: List[OutputWriter], pre_execution_hook: ExecutionHook | None = None, train_evaluation: Evaluation | None = None, test_evaluation: Evaluation | None = None, parameter_input: ParameterInput | None = None, persistence: ModelPersistence | None = None, fit_kwargs: Dict[str, Any] | None = None, predict_kwargs: Dict[str, Any] | None = None)

Bases: Callback

An experiment that trains and evaluates a machine learning model on a specific data set using cross validation or separate training and test sets.

class ExecutionHook

Bases: ABC

An abstract base class for all operations that may be executed before or after an experiment.

abstract execute()

Must be overridden by subclasses in order to execute the operation.

run()

Runs the experiment.

train_and_evaluate(meta_data: MetaData, data_split: DataSplit, train_x, train_y, test_x, test_y)

Trains a model on a training set and evaluates it on a test set.

Parameters:

  • meta_data – The meta-data of the training data set

  • data_split – Information about the split of the available data that should be used for training and evaluating the model

  • train_x – The feature matrix of the training examples

  • train_y – The output matrix of the training examples

  • test_x – The feature matrix of the test examples

  • test_y – The output matrix of the test examples

class mlrl.testbed.experiments.GlobalEvaluation(prediction_type: PredictionType, output_writers: List[OutputWriter])

Bases: Evaluation

Obtains and evaluates predictions from a previously trained global model.

predict_and_evaluate(problem_type: ProblemType, meta_data: MetaData, data_split: DataSplit, data_type: DataType, train_time: float, learner, x, y, **kwargs)

Must be implemented by subclasses in order to obtain and evaluate predictions for given query examples from a previously trained model.

Parameters:

  • problem_type – The type of the machine learning problem

  • meta_data – The meta-data of the data set

  • data_split – The split of the available data, the predictions and ground truth correspond to

  • data_type – Specifies whether the predictions and ground truth correspond to the training or test data

  • train_time – The time needed to train the model

  • learner – The learner, the predictions should be obtained from

  • x – A numpy.ndarray, scipy.sparse.spmatrix or scipy.sparse.sparray, shape (num_examples, num_features), that stores the feature values of the query examples

  • y – A numpy.ndarray, scipy.sparse.spmatrix or scipy.sparse.sparray, shape (num_examples, num_outputs), that stores the ground truth of the query examples

  • kwargs – Optional keyword arguments to be passed to the model when obtaining predictions

class mlrl.testbed.experiments.IncrementalEvaluation(prediction_type: PredictionType, output_writers: List[OutputWriter], min_size: int, max_size: int, step_size: int)

Bases: Evaluation

Repeatedly obtains and evaluates predictions from a previously trained ensemble model, e.g., a model consisting of several rules, using only a subset of the ensemble members with increasing size.

predict_and_evaluate(problem_type: ProblemType, meta_data: MetaData, data_split: DataSplit, data_type: DataType, train_time: float, learner, x, y, **kwargs)

Must be implemented by subclasses in order to obtain and evaluate predictions for given query examples from a previously trained model.

Parameters:

  • problem_type – The type of the machine learning problem

  • meta_data – The meta-data of the data set

  • data_split – The split of the available data, the predictions and ground truth correspond to

  • data_type – Specifies whether the predictions and ground truth correspond to the training or test data

  • train_time – The time needed to train the model

  • learner – The learner, the predictions should be obtained from

  • x – A numpy.ndarray, scipy.sparse.spmatrix or scipy.sparse.sparray, shape (num_examples, num_features), that stores the feature values of the query examples

  • y – A numpy.ndarray, scipy.sparse.spmatrix or scipy.sparse.sparray, shape (num_examples, num_outputs), that stores the ground truth of the query examples

  • kwargs – Optional keyword arguments to be passed to the model when obtaining predictions