Data Preprocessing¶
Libraries Import¶
In [1]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
Data Import¶
In [2]:
# read in the csv file
train = pd.read_csv("football_train.csv", index_col = 0)
In [3]:
# check if the file has been correctly read in
train.head()
Out[3]:
In [4]:
train.shape
Out[4]:
In [5]:
# standardize the numeric columns
n = train.shape[1]
for i in range(n - 1):
train.iloc[:, i] = (train.iloc[:, i] - np.mean(train.iloc[:, i])) / np.std(train.iloc[:, i])
In [6]:
train.head()
Out[6]:
Correlation matrix and correlation heatmap¶
In [7]:
train.corr().iloc[:, n-1]
Out[7]:
In [8]:
plt.figure(figsize=(16,16))
sns.heatmap(train.corr(), annot = True)
plt.xlim([0, n])
plt.ylim([0, n])
plt.title("correlation heatmap: numeric variables")
plt.show()
In [9]:
indices = []
for i in range(n-1):
if (abs(train.corr().iloc[:, n-1]) > 0.35)[i]:
indices.append(i)
Validation Methods¶
Train-Test-Split Method¶
Suppose that we want to predict the variable Wins of each team based on the predictors in the dataframe.
We can choose to use the entire dataset as the testing set, but we then cannot have a measure of how good predictions the model can make on future/unseen data.
Therefore, to enhance the predictive ability of the model, we split the dataset into a training set and a testing set:
- training set: used to build the model (i.e., obtain the coefficients of the model)
- testing set: used to test how good the model is (i.e., to test if the model is overfitting data observed)
Since there are 380 observations in the data frame train, we can use the 70%/30% split method, where 70% of the entries will serve as the training set while the other 30% will be the testing set.
There are two methods to split the dataset:
- using the
train_test_splitfunction inscikit-learnmodule - using the random number generator
samplefunction inrandommodule
train_test_split() function¶
In [10]:
# import the function from module
from sklearn.model_selection import train_test_split
In [11]:
X = train.iloc[:, range(0, n-1)]
y = train.iloc[:, n-1]
In [12]:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 42)
print("size of the training set is %d, size of the testing set is %d." %(X_train.shape[0], X_test.shape[0]))
sample() function¶
Note: this method takes relatively more steps than using the train_test_split() function.
In [13]:
import random
start = 0
end = X.shape[0]
size = X.shape[0]
# obtain the indices of training set
train_indices = random.sample(range(start, end), int(size * 0.7))
In [14]:
# to obtain the indices for testing set
test_indices = list(set(range(start, end)) - set(train_indices))
In [15]:
train_X = X.iloc[train_indices, :]
train_y = y.iloc[train_indices]
test_X = X.iloc[test_indices, :]
test_y = y.iloc[test_indices]
In [16]:
print("size of the training set is %d, size of the testing set is %d." %(train_X.shape[0], test_X.shape[0]))
In [17]:
# check if there is any duplicates in two dataframes
print(test_indices in train_indices)
print(train_indices in test_indices)
Variance-Bias Tradeoff¶
- Variance: the variance of the predicted values.
- Bias: the expected difference between the predicted value and the actual value for the new values of x.
In [18]:
import statsmodels.api as sm # another package for linear model
import statsmodels.formula.api as smf
In [19]:
# create a dataframe for both the training and the testing
df_train = pd.concat([X_train, y_train], axis = 1)
# linear regression model using statsmodel
model1 = smf.ols("Wins ~ Yards + TurnOversLost + FirstDowns + InterceptionsThrown + OppYardsGainedRushing", data = df_train)
results = model1.fit()
print(results.summary())
In [20]:
model2 = smf.ols("Wins ~ Yards + TurnOversLost + FirstDowns", data = df_train)
results2 = model2.fit()
print(results2.summary())
In [21]:
# true test set
test_x = pd.read_csv("football_test.csv", index_col = 0)
test_y = pd.read_csv("football_answers.csv")
In [22]:
# standardize the features
n = test_x.shape[1]
for i in range(n - 1):
test_x.iloc[:, i] = (test_x.iloc[:, i] - np.mean(test_x.iloc[:, i])) / np.std(test_x.iloc[:, i])
In [23]:
from sklearn.linear_model import LinearRegression
In [24]:
model1_indices = [0, 2, 4, 8, 25]
model2_indices = [0, 2, 4]
In [25]:
model1 = LinearRegression().fit(X_train.iloc[:, model1_indices], y_train)
pred1 = model1.predict(X.iloc[:, model1_indices])
model2 = LinearRegression().fit(X_train.iloc[:, model2_indices], y_train)
pred2 = model2.predict(X.iloc[:, model2_indices])
In [26]:
model1.coef_
Out[26]:
In [27]:
model2.coef_
Out[27]:
In [28]:
nrows = X.shape[0]
plt.figure(figsize = (16, 6))
plt.subplot(1, 2, 1)
plt.scatter(y, pred1)
plt.xlabel("true values")
plt.ylabel("predicted values")
plt.subplot(1, 2, 2)
plt.scatter(y, pred2)
plt.xlabel("true values")
plt.ylabel("predicted values")
plt.show()
Now we will fit multiple models on the training set and see the model performance on the dataset:
Building Linear Regression Models with different complexities¶
In [29]:
indices
Out[29]:
In [30]:
# to generate all possible combinations of the indices
import itertools
In [31]:
def createMLRModel(train_X, train_y, test_X, indices, complexity):
'''
This function creates a Multiple Linear Regression Model based on the input
dataframes, indices of the predictors, and the complexity of the model.
train_X: training set of the predictors
train_y: training set of the outcome
test_X: testing set of the predictors
indices: index of each predictor used in the modeling
complexity: number of predictors required
return: predictions - a list of list that contains the predictions of each model
'''
predictions = []
for subset in itertools.combinations(indices, complexity):
model = LinearRegression().fit(train_X.iloc[:, list(subset)], train_y)
pred = model.predict(test_X.iloc[:, list(subset)])
predictions.append(pred.reshape(1,-1))
return predictions
Create regression models and obtain the predictions
In [32]:
predictions = {}
for i in range(1, 10):
predictions[i] = createMLRModel(X_train, y_train, test_x, indices, i)
print("%d MLR models built with complexity = %d" %(len(predictions[i]), i))
Calculate the average value of predictions at each complexity level
In [33]:
averages = {}
for i in range(1, 10):
averages[i] = list(np.mean(predictions[i], axis = 0)[0])
Calculate the bias of predictions at each complexity level
In [34]:
test_y = test_y.loc[:, "Wins"]
In [35]:
biases = {}
for i in range(1, 10):
biases[i] = np.mean(np.abs(averages[i] - test_y))
Calculate the average value of variance in predictions at each complexity level
In [36]:
variances = {}
for i in range(1, 10):
variances[i] = np.mean(np.var(predictions[i]))
Visualize the decrease in bias as the complexity level increases
In [37]:
nrows = test_y.shape[0]
plt.figure(figsize = (16, 9))
plt.subplots_adjust(hspace = 0.5)
for i in range(9):
plt.subplot(3, 3, i+1)
plt.scatter(test_y, averages[i+1])
plt.xlabel("true values")
plt.ylabel("average predictions")
if i == 0:
plt.title("SLR model (1 feature)")
else:
plt.title("MLR with %d features" %(i+1))
Bias vs. model complexity
In [38]:
plt.plot(list(biases.keys()), list(biases.values()),"-o")
plt.xlabel("number of features")
plt.ylabel("bias in the predictions")
plt.show()
Variance vs. model complexity
In [39]:
plt.plot(list(variances.keys()), list(variances.values()),"-o", color = "orange")
plt.xlabel("number of features")
plt.ylabel("variance in the predictions")
plt.show()
Bias vs. Variance as model complexity increases
In [40]:
plt.plot(list(biases.keys()), list(biases.values()),"-o")
plt.plot(list(variances.keys()), list(variances.values()),"-o")
plt.xlabel("number of features")
plt.ylabel("bias or variance")
plt.legend(["bias", "variance"])
plt.title("bias vs. variance tradeoff")
plt.show()
© Kaixin Wang, updated February 2020