Class 17: Data-driven modeling IV¶

This notebook is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Load packages¶

Select a random seed for notebook reproducibility.

Recap of model selection¶

Load the Filip dataset.

What are we trying to do?¶

We considered the problem of selecting a model that best describes the data in the Filip dataset.

For our space of models, we considered different cut-offs for the polynomial summation:

In principle, there is an infinite number of models to choose from, as we can always include more terms in the summation. In practice, we cannot have more terms than datapoints, but even with that restriction, we still have a large number of models to choose from.

How do we decide which model is best?

We want to avoid overfitting, which would mean we're starting to fit to the statistical noise inherent in this data. This means we need to strike a balance between explanatory/predictive power and generalizability. We use cross-validation to help us compare and rank the models according to how well they achieve this balance.

Cross-validation¶

What is it?¶

Cross-validation is a technique used to assess how well a model's predictions generalize to an independent data set. There are different ways to do this. A popular flavor of cross-validation (especially among data scientists) is called k-fold cross-validation. The basic idea is to estimate how robust your model is by systematically removing different chunks (the "folds") of the dataset, repeating the fitting process, then testing its predictive power on the folds. The schematic below illustrates the procedure.

The above example illustrates a 5-fold, or $k=5$, cross-validation. Each fold will act as a testing set, with the remaining $k-1$ folds used to train the model. You fit the model using the training set and try to predict values in the current fold, which you quantify by calculating a score, such as the mean-squared prediction error (MSE). The important feature here is that you are trying to predict data that was not* used in the training stage*. When cross-validation is completed, you will have a history of scores for each model. The average score for each model can then be compared as a means for ranking and selecting a final model.

Why do we use it?¶

In an ideal setting, you would use your current dataset to train a predictive model, and then test it out by going out and collecting more data and seeing how well the model predicts these new data points. However, there are many reasons in practice that this can be difficult to do. It can be expensive and/or technically difficult to gather new data. It may not be possible to gather new data relevant to the problem at hand. So, we can use methods like cross-validation to overcome some of these problems. Cross-validation also helps us to guard against overfitting. If a model has been overfit, then it's predictions will be poor for any data points that are not part of the training set.

When should we use it?¶

Anytime you need to assess the performance and generalizability of a model on unavailable data and a theoretical analysis of the problem is not available. In addition to select between different models, you can also use cross-validation to optimize a single model's internal parameters. This method is an indispensible part of the computational scientist's toolbox.

Cross-validation code¶

Bootstrapping/resampling¶

What is it?¶

Bootstrapping is a method for estimating statistical properties by sampling with replacement from the distribution of the available dataset. This method rests on the assumption that the available dataset distribution reasonably approximates the "true" statistical distribution.

A nice visualization of the bootstrapping method is available on Seeing Theory: https://seeing-theory.brown.edu/frequentist-inference/index.html#section3

Why do we use it?¶

We use bootstrapping when we want to analyze how fluctuations impact statistical properties, for example the mean, median, variance, and so on. It is particularly useful if we do not know the underlying "true" data distribution, or if there are important outliers in our dataset that would be missed by idealizing our distribution. We can use it to construct confidence intervals, which can be used to measure the standard error of a statistical property. It can also be used for statistical inference, for example for running hypothesis tests using p-values.

When should we use it?¶

If you want to know how much variation there is in a statistical property and you do not know the "true" underlying data distribution. Even if you do know the underlying distribution, it can be convenient to use bootstrapping in a re-usable workflow that may be applied to different datasets. If you make assumptions about the underlying dataset, you will have to change your code each time the dataset changes. If you use bootstrapping, then you no longer need to do this.

Bootstrapping code¶

Difference between cross-validation and bootstrapping¶

Cross-validation and bootstrapping share some similar features. Both involve random partitioning of data in some way, and let us see how statistical fluctuations impact our results. Despite these, they are distinct and used for different purposes. We used both in our analysis of the Filip dataset.

• Cross-valdiation: We used this to score the predictive power of different polynomial summation models. The output of the cross-validation runs was a history of cross-validation scores for each model, representing different partitionings of the training and test datasets. The average of these runs for a single model yields the cross-validated score for the model, which we use to rank and compare model performance.

• Bootstrapping: We used this to compute the confidence interval of the cross-validated score. This was possible because our cross-validation runs yielded a statistical distribution of scores. If (and this is important!) our cross-validated scores were normally distributed, then the 95% confidence interval of the cross-validated score would be given by the formula $\bar{x}\pm{}(\sigma_{\bar{x}}\times{}1.96)$, where $\sigma_{\bar{x}}$ is the standard error:

  $$\sigma_{\bar{x}}=\dfrac{\sigma{}}{\sqrt{n}}$$

  However, as we saw, some of our distributions are not normally distributed, and the confidence interval formula does not hold. Generally, the confidence interval narrows as more samples are included in the average. Finding the 95% confidence interval for our statistical averages lets us assess which of our scores are statistically different from one another.

Visualizing and reporting the scores¶

There are two choices for what to use as the error bars when we visualize and report on the statistical distribution of the cross-validated scores:

1. Use the error bars to represent the overall spread of the cross-validation runs. To show this, you might set the limits of the error bars to be the upper and lower bounds for the middle 95% of the data. This does not require any bootstrapping.

2. Use the error bars to represent the overall accuracy of the cross-validated scores. To show this, you would compute a confidence interval for the cross-validated scores using bootstrapping.

We compute the error bars for the overall accuracy of the cross-validated scores first:

To look at the spread of the cross-validation scores, we just need to compute the lower and upper bounds for the middle 95% of the values in the filip_cv_scores_dist_df data frame. The code below finds those percentiles and joins them into the data frame.

The final data frame is as follows:

We visualize our mean-squared error scores, this time with proper error bars representing the 95% confidence interval for the mean scores:

The error bars are very tiny, smaller than the size of the points.

The visualization below shows the mean-squared error scores along with error bars that represent the spread of the cross-validation results (specifically, the middle 95%).