The benchmarking
modules allows you to easily orchestrate benchmarking experiments in which you want to
compare the performance of one or more algorithms over one or more datasets and benchmark configurations.
Benchmarking as an endeavour in general is very easy to get wrong, giving false conclusions about estimator performance - see this 2022 research from Princeton for numerous examples of such mistakes in peer reviewed academic papers as evidence of this.
sktime
's benchmarking
module is designed to provide benchmarking functionality while enforcing best
practices and structure to help users avoid making mistakes (such as data leakage, etc.) which invalidate
their results. The benchmarking
module is designed for easy usage in mind, as such it interfaces
directly with sktime
objects and classes. Previously developed estimator should be usable as they are without
alterations.
This notebook demonstrates usage of the benchmarking
module.
from sktime.benchmarking.forecasting import ForecastingBenchmark
from sktime.datasets import load_airline
from sktime.forecasting.naive import NaiveForecaster
from sktime.performance_metrics.forecasting import MeanSquaredPercentageError
from sktime.split import ExpandingWindowSplitter
In this example we are comparing forecasting estimators.
benchmark = ForecastingBenchmark()
We add different competing estimators to the benchmark instance. All added estimators will be automatically ran through each added benchmark tasks, and their results compiled.
benchmark.add_estimator(
estimator=NaiveForecaster(strategy="mean", sp=12),
estimator_id="NaiveForecaster-mean-v1",
)
benchmark.add_estimator(
estimator=NaiveForecaster(strategy="last", sp=12),
estimator_id="NaiveForecaster-last-v1",
)
These are the prediction/validation tasks over which every estimator will be tested and their results compiled.
The exact arguments for a benchmarking task depend on the whether the objective is forecasting, classification, etc., but generally they are similar. The following are the required arguments for defining a forecasting benchmark task.
Define cross-validation split regimes, using standard sktime
objects.
cv_splitter = ExpandingWindowSplitter(
initial_window=24,
step_length=12,
fh=12,
)
Define performance metrics on which to compare estimators, using standard sktime
objects.
scorers = [MeanSquaredPercentageError()]
Define dataset loaders, which are callables (functions) which should return a dataset. Generally
this is a callable which returns a dataframe containing the entire dataset. One can use
the sktime
defined datasets, or define their own. Something as simple as the following
example will suffice:
def my_dataset_loader():
return pd.read_csv("path/to/data.csv")
The datasets will be loaded when running the benchmarking tasks, ran through the cross-validation regime(s) and subsequently the estimators will be tested over the dataset splits.
dataset_loaders = [load_airline]
Use the previously defined objects to add tasks to the benchmark instance. Optionally use loops etc. to easily setup multiple benchmark tasks reusing arguments.
for dataset_loader in dataset_loaders:
benchmark.add_task(
dataset_loader,
cv_splitter,
scorers,
)
Note that run
won't rerun tasks it already has results for, so adding a new
estimator and running run
again will only run tasks for that new estimator.
results_df = benchmark.run("./forecasting_results.csv")
results_df.T
0 | 1 | |
---|---|---|
validation_id | [dataset=load_airline]_[cv_splitter=ExpandingW... | [dataset=load_airline]_[cv_splitter=ExpandingW... |
model_id | NaiveForecaster-last-v1 | NaiveForecaster-mean-v1 |
runtime_secs | 0.061472 | 0.081733 |
MeanSquaredPercentageError_fold_0_test | 0.024532 | 0.049681 |
MeanSquaredPercentageError_fold_1_test | 0.020831 | 0.0737 |
MeanSquaredPercentageError_fold_2_test | 0.001213 | 0.05352 |
MeanSquaredPercentageError_fold_3_test | 0.01495 | 0.081063 |
MeanSquaredPercentageError_fold_4_test | 0.031067 | 0.138163 |
MeanSquaredPercentageError_fold_5_test | 0.008373 | 0.145125 |
MeanSquaredPercentageError_fold_6_test | 0.007972 | 0.154337 |
MeanSquaredPercentageError_fold_7_test | 0.000009 | 0.123298 |
MeanSquaredPercentageError_fold_8_test | 0.028191 | 0.185644 |
MeanSquaredPercentageError_fold_9_test | 0.003906 | 0.184654 |
MeanSquaredPercentageError_mean | 0.014104 | 0.118918 |
MeanSquaredPercentageError_std | 0.011451 | 0.051265 |