mlrl.testbed.experiments.output.evaluation.measures module¶

Author Michael Rapp (michael.rapp.ml@gmail.com)

Provides classes for implementing evaluation measures.

class mlrl.testbed.experiments.output.evaluation.measures.AggregationMeasure(option_key: str, name: str, aggregation_function: AggregationFunction, can_be_averaged: bool = True, **kwargs)¶

Bases: OutputValue

An aggregation measure that aggregates evaluation results for several experiments.

AggregationFunction¶: alias of Callable[[list[float], bool], Iterable[float]]

aggregate(values: list[float], smaller_is_better: bool) → Iterable[float]¶

Applies the aggregation function to given evaluation results.

Parameters:

values – A list that stores the values to be aggregated
smaller_is_better – True, if smaller values are better than larger ones, False otherwise

Returns:

An iterable that provides access to the aggregated values

class mlrl.testbed.experiments.output.evaluation.measures.Measure(option_key: str, name: str, evaluation_function: EvaluationFunction, smaller_is_better: bool = False, percentage: bool = True, **kwargs)¶

Bases: OutputValue

An evaluation measure.

EvaluationFunction¶: alias of Callable[[Any, Any], float]

UNIT_SECONDS = 'seconds'¶

evaluate(ground_truth: Any, predictions: Any) → float¶

Applies the evaluation function to given predictions and the corresponding ground truth.

Parameters:

ground_truth – A numpy.ndarray, scipy.sparse.spmatrix or scipy.sparse.sparray, shape (num_examples, num_outputs), that stores the ground truth
predictions – A numpy.ndarray, scipy.sparse.spmatrix or scipy.sparse.sparray, shape (num_examples, num_outputs), that stores the predictions to be evaluated

Returns:

An evaluation score in [0, 1]

static is_smaller_better(measure_name: str) → bool¶

Returns whether smaller values are better than larger values according to a measure, depending on its name.

Parameters:: measure_name – Name of the measure
Returns:: True, if smaller values are better than larger ones, False otherwise