Course: Building an Effective ML Workflow with scikit-learn¶
Outline:¶
- Review of the basic Machine Learning workflow
- Encoding categorical data
- Using ColumnTransformer and Pipeline
- Recap
- Encoding text data
Part 1: Review of the basic Machine Learning workflow¶
Check your scikit-learn version:
In [1]:
import sklearn
sklearn.__version__
Out[1]:
Load 10 rows from the famous Titanic dataset:
In [2]:
import pandas as pd
df = pd.read_csv('http://bit.ly/kaggletrain', nrows=10)
Basic terminology:
- "Survived" is the target column
- Target is categorical, thus it's a classification problem
- All other columns are possible features
- Each row is an observation, and represents a passenger
- This is our training data because we know the target values
In [3]:
df
Out[3]:
We want to use "Parch" and "Fare" as initial features:
- "Parch" is the number of parents or children aboard with each passenger
- "Fare" is the amount they paid
- Both are numeric
Define X and y:
- X is the feature matrix
- y is the target
In [4]:
X = df[['Parch', 'Fare']]
X
Out[4]:
In [5]:
y = df['Survived']
y
Out[5]:
Check the object shapes:
- X is a pandas DataFrame with 2 columns, thus it has 2 dimensions
- y is a pandas Series, thus it has 1 dimension
In [6]:
X.shape
Out[6]:
In [7]:
y.shape
Out[7]:
Create a model object:
- Set the "solver" to increase the likelihood that we will all get the same results
- Set the "random_state" for reproducibility
In [8]:
from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression(solver='liblinear', random_state=1)
Evaluate the model using cross-validation:
- Our goal is to simulate model performance on future data so that we can choose between models
- Evaluation metric is classification accuracy
- "cross_val_score" does the dataset splitting, training, predictions, and evaluation
- Your results may differ based on your scikit-learn version
- We can't take these results seriously because the dataset is tiny
In [9]:
from sklearn.model_selection import cross_val_score
cross_val_score(logreg, X, y, cv=3, scoring='accuracy').mean()
Out[9]:
Train the model on the entire dataset:
In [10]:
logreg.fit(X, y)
Out[10]:
Read in a new dataset for which we don't know the target values:
In [11]:
df_new = pd.read_csv('http://bit.ly/kaggletest', nrows=10)
df_new
Out[11]:
Define X_new to have the same columns as X:
In [12]:
X_new = df_new[['Parch', 'Fare']]
X_new
Out[12]:
Use the trained model to make predictions for X_new:
In [13]:
logreg.predict(X_new)
Out[13]:
Part 2: Encoding categorical data¶
We want to use "Embarked" and "Sex" as additional features:
- "Embarked" is the port they embarked from
- "Sex" is male or female
- They are unordered categorical features
- They can't be directly passed to the model because they aren't numeric
In [14]:
df
Out[14]:
Encode "Embarked" using one-hot encoding:
- This is the same as "dummy encoding"
- Outputs a sparse matrix, which is more efficient and performant when most values in a matrix are zeros
- Use two brackets around "Embarked" to pass a DataFrame instead of a Series
In [15]:
from sklearn.preprocessing import OneHotEncoder
ohe = OneHotEncoder()
ohe.fit_transform(df[['Embarked']])
Out[15]:
Ask for a dense (not sparse) matrix so that we can examine the encoding:
- There are 3 columns because there were 3 unique values in "Embarked"
- Each row contains a single 1
- 100 means "C", 010 means "Q", 001 means "S"
- The categories are listed in alphabetical order in the "categories_" attribute
- You can think of "categories_" as the column headings for the matrix
- From each of the three features, the model can learn the relationship between the target value and whether or not a given passenger embarked at that port
In [16]:
ohe = OneHotEncoder(sparse=False)
ohe.fit_transform(df[['Embarked']])
Out[16]:
In [17]:
ohe.categories_
Out[17]:
What's the difference between "fit" and "transform"?
- OneHotEncoder is a "transformer", meaning its role is data transformations
- Transformers usually have a "fit" method and always have a "transform" method
- For all transformers: "fit" is when they learn something, and "transform" is when they use what they learned to do the transformation
- For OneHotEncoder: "fit" is when it learns the categories, and "transform" is when it creates the matrix using those categories
- If you are going to "fit" and "transform", then you should do it in a single step using "fit_transform"
Encode "Embarked" and "Sex" at the same time:
- First 3 columns represent "Embarked" and last two columns represent "Sex"
- For the "Sex" columns: 10 means "female", 01 means "male"
In [18]:
ohe.fit_transform(df[['Embarked', 'Sex']])
Out[18]:
In [19]:
ohe.categories_
Out[19]:
How could we include "Embarked" and "Sex" in the model along with "Parch" and "Fare"?
- Stack the 2 numeric features side-by-side with the 5 encoded columns, and then train the model with all 7 columns
- However, we would need to repeat the same process (encoding and stacking) with the new data before making predictions
- Doing this manually is inefficient and error-prone, and the complexity will only increase as you preprocess additional columns
Part 3: Using ColumnTransformer and Pipeline¶
Goals:
- Use ColumnTransformer to make it easy to apply different preprocessing to different columns
- Use Pipeline to make it easy to apply the same workflow to training data and new data
Create a list of columns and use that to update X:
In [20]:
cols = ['Parch', 'Fare', 'Embarked', 'Sex']
In [21]:
X = df[cols]
X
Out[21]:
Create an instance of OneHotEncoder with the default options:
In [22]:
ohe = OneHotEncoder()
Create a ColumnTransformer:
- First argument (a tuple) specifies that we want to one-hot encode the "Embarked" and "Sex" columns
- "remainder" argument specifies that we want to keep all other columns in the final output (without modifying them)
In [23]:
from sklearn.compose import make_column_transformer
ct = make_column_transformer(
(ohe, ['Embarked', 'Sex']),
remainder='passthrough')
Perform the transformation:
- Output contains 7 columns in this order: 3 columns for "Embarked", 2 for "Sex", 1 for "Parch", and 1 for "Fare"
- Column order is the order in which you listed them in the ColumnTransformer followed by any you passthrough
In [24]:
ct.fit_transform(X)
Out[24]:
Use Pipeline to chain together sequential steps:
- Step 1 is data preprocessing using ColumnTransformer
- Step 2 is model building using LogisticRegression
In [25]:
from sklearn.pipeline import make_pipeline
pipe = make_pipeline(ct, logreg)
Fit the Pipeline:
- Step 1: X gets transformed from 4 columns to 7 columns by ColumnTransformer
- Step 2: LogisticRegression model gets fit, thus it learns the relationship between those 7 features and the y values
- Step 1 is assigned the name "columntransformer" (all lowercase), and step 2 is assigned the name "logisticregression"
In [26]:
pipe.fit(X, y)
Out[26]:
This is what happens "under the hood" when you fit the Pipeline:
In [27]:
logreg.fit(ct.fit_transform(X), y)
Out[27]:
You can select the steps of a Pipeline by name in order to inspect them:
- These are the 7 coefficients of the logistic regression model
In [28]:
pipe.named_steps.logisticregression.coef_
Out[28]:
Update X_new to have the same columns as X:
In [29]:
X_new = df_new[cols]
X_new
Out[29]:
Use the fitted Pipeline to make predictions for X_new:
In [30]:
pipe.predict(X_new)
Out[30]:
This is what happens "under the hood" when you make predictions using the Pipeline:
- It uses "transform" rather than "fit_transform" so that the exact encoding scheme learned from the training data (during the "fit" step) will be applied to the new data
In [31]:
logreg.predict(ct.transform(X_new))
Out[31]:
Recap¶
This is all of the code that is necessary to recreate our workflow up to this point:
- You can copy/paste this code from http://bit.ly/basic-pipeline
- There are no calls to "fit_transform" or "transform" because all of that functionality is encapsulated by the Pipeline
In [32]:
import pandas as pd
from sklearn.preprocessing import OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.compose import make_column_transformer
from sklearn.pipeline import make_pipeline
In [33]:
cols = ['Parch', 'Fare', 'Embarked', 'Sex']
In [34]:
df = pd.read_csv('http://bit.ly/kaggletrain', nrows=10)
X = df[cols]
y = df['Survived']
In [35]:
df_new = pd.read_csv('http://bit.ly/kaggletest', nrows=10)
X_new = df_new[cols]
In [36]:
ohe = OneHotEncoder()
In [37]:
ct = make_column_transformer(
(ohe, ['Embarked', 'Sex']),
remainder='passthrough')
In [38]:
logreg = LogisticRegression(solver='liblinear', random_state=1)
In [39]:
pipe = make_pipeline(ct, logreg)
pipe.fit(X, y)
pipe.predict(X_new)
Out[39]:
Summary of our ColumnTransformer:
- It selected 2 categorical columns and transformed them, resulting in 5 columns
- It selected 2 numerical columns and did nothing to them, resulting in 2 columns
- It stacked the 7 columns side-by-side
Summary of our Pipeline:
- Step 1 transformed the data from 4 columns to 7 columns using ColumnTransformer
- Step 2 used a LogisticRegression model for fitting and predicting
Comparing Pipeline and ColumnTransformer:
- ColumnTransformer pulls out subsets of columns and transforms them, and then stacks the results side-by-side
- Pipeline is a series of steps that occur in order, and the output of each step passes to the next step
Part 4: Encoding text data¶
We want to use "Name" as an additional feature:
- It can't be directly passed to the model because it isn't numeric
In [40]:
df
Out[40]:
Use CountVectorizer to convert text into a matrix of token counts:
- Use single brackets around "Name" to pass a Series, because CountVectorizer expects 1-dimensional input
- Outputs a document-term matrix containing 10 rows (one for each name) and 40 columns (one for each unique word)
In [41]:
from sklearn.feature_extraction.text import CountVectorizer
vect = CountVectorizer()
dtm = vect.fit_transform(df['Name'])
dtm
Out[41]:
Examine the feature names:
- It found 40 unique words in the "Name" Series after lowercasing the words, removing punctuation, and removing words that were only 1 character long
In [42]:
print(vect.get_feature_names())
Examine the document-term matrix as a DataFrame:
- In each row, CountVectorizer counted how many times each word appeared
- For example, the first row contains 36 zeros and 4 ones (under "braund", "mr", "owen", and "harris")
- This encoding is known as the "Bag of Words" representation
- From each of the 40 features, the model can learn the relationship between the target value and how many times that word appeared in each passenger's name
In [43]:
pd.DataFrame(dtm.toarray(), columns=vect.get_feature_names())
Out[43]:
Update X to include the "Name" column:
In [44]:
cols = ['Parch', 'Fare', 'Embarked', 'Sex', 'Name']
X = df[cols]
X
Out[44]:
Update the ColumnTransformer:
- Add another tuple to specify that CountVectorizer should be applied to the "Name" column
- There are no brackets around "Name" because CountVectorizer expects 1-dimensional input
In [45]:
ct = make_column_transformer(
(ohe, ['Embarked', 'Sex']),
(vect, 'Name'),
remainder='passthrough')
Perform the transformation:
- Output contains 47 columns in this order: 3 columns for "Embarked", 2 for "Sex", 40 for "Name", 1 for "Parch", and 1 for "Fare"
In [46]:
ct.fit_transform(X)
Out[46]:
Update the Pipeline to contain the modified ColumnTransformer:
In [47]:
pipe = make_pipeline(ct, logreg)
Fit the Pipeline and examine the steps:
In [48]:
pipe.fit(X, y)
pipe.named_steps
Out[48]:
Update X_new to include the "Name" column:
In [49]:
X_new = df_new[cols]
Use the fitted Pipeline to make predictions for X_new:
In [50]:
pipe.predict(X_new)
Out[50]: