After completing this week's lecture and tutorial work, you will be able to:
tidymodels
package to predict the class of a test dataset.This worksheet covers parts of the Classification II chapter of the online textbook. You should read this chapter before attempting the worksheet.
### Run this cell before continuing.
library(tidyverse)
library(repr)
library(tidymodels)
options(repr.matrix.max.rows = 6)
source('tests.R')
source('cleanup.R')
Question 0.1 Multiple Choice:
{points: 1}
Before applying k-nearest neighbour to a classification task, we need to scale the data. What is the purpose of this step?
A. To help speed up the knn algorithm.
B. To convert all data observations to numeric values.
C. To ensure all data observations will be on a comparable scale and contribute equal shares to the calculation of the distance between points.
D. None of the above.
Assign your answer to an object called answer0.1
. Make sure your answer is an uppercase letter and is surrounded by quotation marks (e.g. "F"
).
# your code here
fail() # No Answer - remove if you provide an answer
test_0.1()
Question 1.0
{points: 1}
You will recognize a few of the first questions in this worksheet from last week's tutorial. This will help you repeat some of the fundamentals of classification before tackling the later questions in this worksheet, which integrate concepts you learned from this week's material. First, load the file fruit_data.csv
(found in the data folder) from the previous tutorial into your notebook.
mutate()
the fruit_name
column such that it is a factor using as_factor()
.
Assign your data to an object called fruit_data
.
# your code here
fail() # No Answer - remove if you provide an answer
test_1.0()
Let's take a look at the first six observations in the fruit dataset. Run the cell below.
# Run this cell.
slice(fruit_data, 1:6)
Run the cell below, and find the nearest neighbour based on mass and width to the first observation just by looking at the scatterplot (the first observation has been circled for you).
# Run this cell.
options(repr.plot.width=10, repr.plot.height=6)
point1 <- c(192, 8.4)
point2 <- c(180, 8)
point44 <- c(194, 7.2)
fruit_data |>
ggplot(aes(x=mass,
y= width,
colour = fruit_name)) +
labs(x = "Mass (grams)",
y = "Width (cm)",
colour = 'Name of the Fruit') +
geom_point() +
annotate("path",
x=point1[1] + 5*cos(seq(0,2*pi,length.out=100)),
y=point1[2] + 0.1*sin(seq(0,2*pi,length.out=100))) +
annotate("text", x = 183, y = 8.5, label = "1") +
theme(text = element_text(size = 20))
Question 1.1 Multiple Choice:
{points: 1}
Based on the graph generated, what is the fruit_name
of the closest data point to the one circled?
A. apple
B. lemon
C. mandarin
D. orange
Assign your answer to an object called answer1.1
. Make sure your answer is an uppercase letter and is surrounded by quotation marks (e.g. "F"
).
# your code here
fail() # No Answer - remove if you provide an answer
test_1.1()
Question 1.2
{points: 1}
Using mass
and width
, calculate the distance between the first observation and the second observation.
We provide a scaffolding to get you started.
Assign your answer to an object called fruit_dist_2
.
# ... <- fruit_data |>
# slice(1, 2) |> # We use slice to get the first two rows of the fruit dataset
# select(mass, ...) |>
# dist()
# your code here
fail() # No Answer - remove if you provide an answer
test_1.2()
Question 1.3
{points: 1}
Calculate the distance between the first and the 44th observation in the fruit
dataset using the mass
and width
variables.
You can see from the data frame output from the cell below that observation 44 has mass
= 194 g and width
= 7.2 cm.
Assign your answer to an object called fruit_dist_44
.
# Run this cell.
filter(fruit_data, row_number() == 44)
point1 <- c(192, 8.4)
point2 <- c(180, 8)
point44 <- c(194, 7.2)
fruit_data |>
ggplot(aes(x = mass,
y = width,
colour = fruit_name)) +
labs(x = "Mass (grams)",
y = "Width (cm)",
colour = 'Name of the Fruit') +
geom_point() +
annotate("path",
x=point1[1] + 5*cos(seq(0,2*pi,length.out=100)),
y=point1[2] + 0.1*sin(seq(0,2*pi,length.out=100))) +
annotate("text", x = 183, y = 8.5, label = "1") +
annotate("path",
x=point2[1] + 5*cos(seq(0,2*pi,length.out=100)),
y=point2[2] + 0.1*sin(seq(0,2*pi,length.out=100))) +
annotate("text", x = 169, y = 8.1, label = "2") +
annotate("path",
x=point44[1] + 5*cos(seq(0,2*pi,length.out=100)),
y=point44[2]+0.1*sin(seq(0,2*pi,length.out=100))) +
annotate("text", x = 204, y = 7.1, label = "44") +
theme(text = element_text(size = 20))
# your code here
fail() # No Answer - remove if you provide an answer
test_1.3()
What do you notice about your answers from Question 1.2 & 1.3 that you just calculated? Is it what you would expect given the scatter plot above? Why or why not? Discuss this.
Hint: Look at where the observations are on the scatterplot in the cell above this question. What might happen if we changed grams into kilograms to measure the mass?
Question 1.4
{points: 1}
From the distance calculation, we see that observation 1 and 44 have a smaller distance than observation 1 and 2. However, if we look at the scatterplot the distance of the first observation to the second observation appears closer than to the 44th observation.
Which of the following statements is correct?
A. A difference of 12 g in mass between observation 1 and 2 is large compared to a difference of 1.2 cm in width between observation 1 and 44. Consequently, mass will drive the classification results, and width will have less of an effect. Hence, our distance calculation reflects that.
B. If we measured mass in kilograms, then we’d get different classification results.
C. We should standardize the data so that all variables will be on a comparable scale.
D. All of the above.
Assign your answer to an object called answer1.4
. Make sure your answer is an uppercase letter and is surrounded by quotation marks (e.g. "F"
).
# your code here
fail() # No Answer - remove if you provide an answer
answer1.4
test_1.4()
Question 1.5
{points: 1}
Scale and center all the variables of the fruit
dataset and save them as columns in your data table.
Save the dataset object and call it fruit_data_scaled
. Make sure to name the new columns scaled_*
where * is the old column name (e.g. scaled_mass
). Do not drop the unscaled columns (mass
, width
, height
, color_score
).
# .. <- fruit_data |>
# mutate(scaled_mass = scale(mass, center = TRUE),
# scaled_width = scale(..., ...),
# scaled_height = scale(..., ...),
# scaled_color_score = scale(..., ...))
# your code here
fail() # No Answer - remove if you provide an answer
fruit_data_scaled
test_1.5()
Question 1.6
{points: 1}
Let's repeat Question 1.2 and 1.3 with the scaled variables:
After you do this, think about how these distances compared to the distances you computed in Question 1.2 and 1.3 for the same points.
Assign your answers to objects called distance_2
and distance_44
respectively.
# your code here
fail() # No Answer - remove if you provide an answer
distance_2
distance_44
test_1.6()
This worksheet uses functions from the tidymodels
library, which not only allows us to perform K-nearest neighbour classification, but also allows us to evaluate how well our classification worked. In order to ensure that the steps in the worksheet are reproducible, we need to set a seed, i.e., a numerical "starting value," which determines the sequence of random numbers R will generate.
Below in many cells we have included a call to set.seed
. Do not remove these lines of code; they are necessary to make sure the autotesting code functions properly.
The reason we have
set.seed
in so many places is that Jupyter notebooks are organized into cells that can be run out of order. Since things can be run out of order, the exact sequence of random values that is used in each cell is hard to determine, which makes autotesting really difficult. We had two options: either enforce that you only ever run the code by hitting "Restart & Run All" to ensure that we get the same values of randomness each time, or putset.seed
in a lot of places (we chose the latter). One drawback of callingset.seed
everywhere is that the numbers that will be generated won't really be random. For the purposes of teaching and learning, that is fine here. But in your course project and other data analyses outside of this course, you must callset.seed
only once at the beginning of the analysis, so that your random numbers are actually reasonably random.
In this exercise, we will be partitioning fruit_data
into a training (75%) and testing (25%) set using the tidymodels
package. After creating the test set, we will put the test set away in a lock box and not touch it again until we have found the best k-nn classifier we can make using the training set. We will use the variable fruit_name
as our class label.
Question 2.0
{points: 1}
To create the training and test set, first use the initial_split
function to split fruit_data
. Specify you want to use 75% of the data. For the strata
argument, place the variable we want to classify, fruit_name
. Name the object you create fruit_split
.
Next, pass the fruit_split
object to the training
and testing
functions and name your respective objects fruit_train
and fruit_test
.
# Set the seed. Don't remove this!
set.seed(3456)
# Randomly take 75% of the data in the training set.
# This will be proportional to the different number of fruit names in the dataset.
#... <- initial_split(..., prop = ..., strata = ...)
#... <- training(...)
#... <- testing(...)
# your code here
fail() # No Answer - remove if you provide an answer
fruit_train
fruit_test
test_2.0()
Question 2.1
{points: 1}
K-nearest neighbors is sensitive to the scale of the predictors so we should do some preprocessing to standardize them. Remember that standardizing involves centering/shifting (subtracting the mean of each variable) and scaling (dividing by its standard deviation). Also remember that standardization is part of your training procedure, so you can't use your test data to compute the shift / scale values for each variable. Therefore, you must pass only the training data to your recipe to compute the preprocessing steps. This ensures that our test data does not influence any aspect of our model training. Once we have created the standardization preprocessor, we can then later on apply it separately to both the training and test data sets.
For this exercise, let's see if mass
and color_score
can predict fruit_name
.
To scale and center the data, first, pass the vector and the predictors to the recipe
function. Remember to place your vector before your predictors. To scale your predictors, use the step_scale(all_predictors())
function. To center your predictors, use the step_center(all_predictors())
function.
Assign your answer to an object called fruit_recipe
.
set.seed(2020) # DO NOT REMOVE
#... <- recipe(... ~ ... + ... , data = ...) |>
# ... |>
# ...
# your code here
fail() # No Answer - remove if you provide an answer
fruit_recipe
test_2.1()
Question 2.2
{points: 1}
So far, we have split the training and testing datasets as well as preprocessed the data. Now, let's create our K-nearest neighbour classifier with only the training set using the tidymodels
package. First, create the classifier by specifying that we want $K = 3$ neighbors and that we want to use the straight-line distance.
Assign your answer to an object called knn_spec
.
Next, train the classifier with the training data set using the workflow
function. This function allows you to bundle together your pre-processing, modeling, and post-processing requests. Scaffolding is provided below for you.
Assign your answer to an object called fruit_fit
.
set.seed(2020) # DO NOT REMOVE
#... <- nearest_neighbor(weight_func = ..., neighbors = ...) |>
# set_engine(...) |>
# set_mode(...)
#... <- workflow() |>
# add_recipe(...) |>
# add_model(...) |>
# fit(data = ...)
# your code here
fail() # No Answer - remove if you provide an answer
fruit_fit
test_2.2()
Question 2.3
{points: 1}
Now that we have created our K-nearest neighbor classifier object, let's predict the class labels for our test set.
First, pass your fitted model and the test dataset to the predict
function. Then, use the bind_cols
function to add the column of predictions to the original test data.
Assign your answer to an object called fruit_test_predictions
.
set.seed(2020) # DO NOT REMOVE
#... <- predict(... , ...) |>
# ...(...)
# your code here
fail() # No Answer - remove if you provide an answer
fruit_test_predictions
test_2.3()
Question 2.4
{points: 1}
Great! We have now computed some predictions for our test datasets! From glancing at the dataframe above, it looks like most of them are correct, but wouldn't it be interesting if we could find out our classifier's accuracy?
Thankfully, the metrics
function from the tidymodels
package can help us. To get the statistics about the quality of our model, you need to specify the truth
and estimate
arguments. In the truth
argument, you should put the column name for the true values of the response variable. In the estimate
argument, you should put the column name for response variable predictions.
Assign your answer to an object called fruit_prediction_accuracy
.
set.seed(2020) # DO NOT REMOVE
#... <- fruit_test_predictions |>
# ...(truth = ..., estimate = ...)
# your code here
fail() # No Answer - remove if you provide an answer
fruit_prediction_accuracy
test_2.4()
Question 2.5
{points: 1}
Now, let's look at the confusion matrix for the classifier. This will show us a table comparing the predicted labels with the true labels.
A confusion matrix is essentially a classification matrix. The columns of the confusion matrix represent the actual class and the rows represent the predicted class (or vice versa). Shown below is an example of a confusion matrix.
Actual Values | |||
---|---|---|---|
Positive | Negative | ||
Predicted Value | Positive | True Positive | False Positive |
Negative | False Negative | True Negative |
We can create a confusion matrix by using the conf_mat
function. Similar to the metrics
function, you will have to specify the truth
and estimate
arguments.
Assign your answer to an object called fruit_mat
.
set.seed(2020) # DO NOT REMOVE
#... <- fruit_test_predictions |>
# ...(truth = ..., estimate = ...)
# your code here
fail() # No Answer - remove if you provide an answer
fruit_mat
test_2.5()
Question 2.6 Multiple Choice:
{points: 1}
Reading fruit_mat
, how many observations were labelled correctly?
A. 14
B. 12
C. 10
D. 13
Assign your answer to an object called answer2.6
. Make sure your answer is an uppercase letter and is surrounded by quotation marks (e.g. "F"
).
# your code here
fail() # No Answer - remove if you provide an answer
answer2.6
test_2.6()
Question 3.1
{points: 1}
The vast majority of predictive models in statistics and machine learning have parameters that you have to pick. For the past few exercises, we have had to pick the number of neighbours for the class vote, which we have done arbitraily. But, is it possible to make this selection, i.e., tune the model, in a principled way? Ideally, we want to pick the number of neighborurs to maximize the performance of our classifier on data it hasn’t seen yet.
An important aspect of the tuning process is that we can, if we want to, split our training data again, train and evaluate a classifier for each split, and then choose the parameter based on all of the different results. If we just split our training data once, our best parameter choice will depend strongly on the randomness from how this single split was made. Using multiple different splits, we’ll get a more robust estimate of accuracy, which will lead to a more suitable choice of the number of neighbours $K$ to perform well on unseen data.
The idea of training and evaluating models on multiple training data splits times is called "cross-validation". In cross-validation, we split our overall training data into $C$ evenly-sized chunks, and then iteratively use 1 chunk as the validation set and combine the remaining $C−1$ chunks as the training set. The validation set is used in a similar was as the test set, except that the test set is only used once at the end to report model performance whether we use model performance on the validation set to select the model during cross-validation.
We can perform a cross-validation in R using the vfold_cv
function. To use this function, you have to identify the training set as well as specify the v
(the number of folds) and the strata
argument (the label variable).
For this exercise, perform $5$-fold cross-validation.
Assign your answer to an object called fruit_vfold
.
set.seed(2020) # DO NOT REMOVE
#... <- vfold_cv(..., v = ..., strata = ...)
# your code here
fail() # No Answer - remove if you provide an answer
test_3.1()
Question 3.2
{points: 1}
Now perform the workflow analysis again. You can reuse the fruit_recipe
and knn_spec
objects you made earlier. When you are fitting the knn model, use the fit_resamples
function instead of the fit
function for training. This function will allow us to run a cross-validation on each train/validation split we created in the previous question.
Assign your answer to an object called fruit_resample_fit
.
set.seed(2020) # DO NOT REMOVE
#... <- workflow() |>
# add_recipe(...) |>
# add_model(...) |>
# fit_resamples(resamples = ...)
# your code here
fail() # No Answer - remove if you provide an answer
test_3.2()
Question 3.3
{points: 1}
Now that we have ran a cross-validation on each train/validation split, one has to ask, how accurate was the classifier's validation across the folds? We can aggregate the mean and standard error by using the collect_metrics
function. The standard error is essentially a measure of how uncertain we are in the mean value.
Use the collect_metrics
function on the fruit_resample_fit
object and assign your answer to an object called fruit_metrics
.
# your code here
fail() # No Answer - remove if you provide an answer
fruit_metrics
test_3.3()
Using a 5-fold cross-validation, we have established a prediction accuracy for our classifier.
If we had to improve our classifier, we have to change the parameter: number of neighbours, $K$. Since cross-validation helps us evaluate the accuracy of our classifier, we can use cross-validation to calculate an accuracy for each value of $K$ in a reasonable range, and then pick the value of $K$ that gives us the best accuracy.
The great thing about the tidymodels
package is that it provides a very simple syntax for tuning models. Using tune()
, each parameter in the model can be adjusted rather than given a specific value.
Question 4.0
{points: 1}
Create a new K-nearest neighbor model specification but instead of specifying a particular value for the neighbors
argument, insert tune()
.
Assign your answer to an object called knn_tune
.
set.seed(2020) # DO NOT REMOVE
# your code here
fail() # No Answer - remove if you provide an answer
knn_tune
test_4.0()
Question 4.1
{points: 1}
Now, create a workflow()
analysis that combines fruit_recipe
and our new knn_tune
model specification.
Instead of using fit
or fit_resamples
, we will use the tune_grid
function to fit the model for each value in a range of parameter values. For the resamples
argument, input the cross-validation fruit_vfold
model we created earlier. The grid
argument specifies that the tuning should try $X$ amount of values of the number of neighbors $K$ when tuning. For this exercise, use 10 $K$ values when tuning.
Finally, aggregate the mean and standard error by using the collect_metrics
function.
Assign your answer to an object called knn_results
.
set.seed(1234) # set the seed, don't remove this
#... <- workflow() |>
# add_recipe(...) |>
# add_model(...) |>
# tune_grid(resamples = ..., grid = ...) |>
# ...
# your code here
fail() # No Answer - remove if you provide an answer
knn_results
test_4.1()
Question 4.2
{points: 1}
Now, let's find the best value of the number of neighbors.
First, from knn_results
, filter for accuracy
from the .metric
column.
Assign your answer to an object called accuracies
.
Next, create a line plot using the accuracies
dataset with neighbors
on the x-axis and the mean
on the y-axis.
Assign your answer to an object called accuracy_versus_k
.
#... <- knn_results |>
# filter(...)
#... <- ggplot(..., aes(x = ..., y = ...))+
# geom_point() +
# geom_line() +
# labs(x = "Neighbors", y = "Accuracy Estimate") +
# scale_x_continuous(breaks = seq(0, 14, by = 1)) + # adjusting the x-axis
# scale_y_continuous(limits = c(0.4, 1.0)) # adjusting the y-axis
# your code here
fail() # No Answer - remove if you provide an answer
accuracy_versus_k
test_4.2()
From the plot above, we can see that $K = 2$, $3$, or $4$ provides the highest accuracy. Larger $K$ values result in a reduced accuracy estimate. Remember: the values you see on this plot are estimates of the true accuracy of our classifier. Although the $K = 2$, $3$ or $4$ value is higher than the others on this plot, that doesn’t mean the classifier is necessarily more accurate with this parameter value!
Great, now you have completed a full workflow analysis with cross-validation using the tidymodels
package! For your information, we can choose any number of folds and typically, the more we use the better our accuracy estimate will be (lower standard error). However, more folds would mean a greater computation time. In practice, $C$ is chosen to be either 5 or 10.
source('cleanup.R')