Newsgroup Classification¶
In this notebook we will use a larger portion of the 20-Newsgroup data to perform text categorization. We will explore the use of several concepts including sparse matrices and pipleplines. We will also use parameter optimization to do model selection with Naive Bayes and Support Vector Machine classifiers.¶
In [1]:
%pylab inline
In [2]:
import pandas as pd
import numpy as np
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
In [3]:
from sklearn.datasets import fetch_20newsgroups
news = fetch_20newsgroups(subset='all')
# We'll use only the first 3000 documents
n_samples = 3000
X = news.data[:n_samples]
y = news.target[:n_samples]
In [4]:
news.target_names
Out[4]:
In [5]:
### Each document is a message posted on a newsgroup. Below is the text of the first two messages in the data.
X[:2]
Out[5]:
In [6]:
### Class labels for the newsgroups
print(set(y))
In [7]:
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=11)
In [8]:
### As part of preprocessing, we will remove the stop word
stopwords = pd.read_csv('stopwords_en.txt', header=None)
stopwords
Out[8]:
In [9]:
stop_words = list(stopwords.values.ravel())
In [10]:
# TfidfVectorizer performs tokenization, removes stop words, and performs tfxidf transformation
tfidf = TfidfVectorizer(stop_words=stop_words, token_pattern=r"\b[a-z0-9_\-\.]+[a-z][a-z0-9_\-\.]+\b", norm=None)
In [11]:
X_tfidf = tfidf.fit_transform(X_train)
In [12]:
X_tfidf.shape
Out[12]:
In [13]:
X_test_tfidf = tfidf.transform(X_test)
In [14]:
X_test_tfidf.shape
Out[14]:
Note that the transformed data is stored in a sparse matrix (which is much more efficient for large data sets)¶
In [15]:
X_tfidf
Out[15]:
In [16]:
print(X_tfidf[:2])
In [17]:
# It's possible (though not usually necessary) to convert the matrix into a "dense" matrix
newX = X_tfidf.todense()
In [18]:
newX.shape
Out[18]:
In [19]:
np.set_printoptions(linewidth=120, edgeitems=12)
print(newX[:10])
In [20]:
print(newX.sum(axis=1))
Fortunately, scikit-learn modules handle sparse matrices natively. So, the sparse matrices, such as X_tfidf above, an be directly passed to scikit-learn models.¶
Using Pipelines¶
Pipelines allow us to create machine learning workflows that combine different tasks. For example, preprocessing tasks can be combined with model creation. In the following examples, we'll use a pipeline to integrate the tokenization and TFIDF transformation preprocessing steps into our classification model. The full pipeline can then be used in training and evaluation of models.¶
In [21]:
### Lets' set up a pipeline to perform preprocessng of the newsgroup data and
### classification of the documents using Multiomial Naive Bayes
clf = Pipeline([
('vect', TfidfVectorizer(
stop_words=stop_words,
token_pattern=r"\b[a-z0-9_\-\.]+[a-z][a-z0-9_\-\.]+\b",
)),
('nb', MultinomialNB(alpha=0.01)),
])
In [22]:
### We can use this function ot do cross-validation
from sklearn.model_selection import cross_val_score, KFold
from scipy.stats import sem
def evaluate_cross_validation(clf, X, y, K):
# create a k-fold cross validation iterator of K folds
cv = KFold(n_splits=K, random_state=0, shuffle=True)
# by default the score used is the one returned by score method of the estimator (accuracy)
scores = cross_val_score(clf, X, y, cv=cv)
print(scores)
print("Mean score: %.3f (+/-%.3f)" % (np.mean(scores), sem(scores)))
Note that the full pipeline is beign passed as the model to the cross-validation function¶
In [23]:
evaluate_cross_validation(clf, X_train, y_train, 5)
To optimize model parameters individually (in this case Naive Bayes smoothing parameter alpha), we can use our "calc_params' function used in earlier examples.¶
In [24]:
def calc_params(X, y, clf, param_values, param_name, K):
# Convert input to Numpy arrays
X = np.array(X)
y = np.array(y)
# initialize training and testing score arrays with zeros
train_scores = np.zeros(len(param_values))
test_scores = np.zeros(len(param_values))
# iterate over the different parameter values
for i, param_value in enumerate(param_values):
# set classifier parameters
clf.set_params(**{param_name:param_value})
# initialize the K scores obtained for each fold
k_train_scores = np.zeros(K)
k_test_scores = np.zeros(K)
# create KFold cross validation
cv = KFold(n_splits=K, shuffle=True, random_state=0)
# iterate over the K folds
j = 0
for train, test in cv.split(X):
# fit the classifier in the corresponding fold
# and obtain the corresponding accuracy scores on train and test sets
clf.fit(X[train], y[train])
k_train_scores[j] = clf.score(X[train], y[train])
k_test_scores[j] = clf.score(X[test], y[test])
j += 1
# store the mean of the K fold scores
train_scores[i] = np.mean(k_train_scores)
test_scores[i] = np.mean(k_test_scores)
print(param_name, '=', param_value, "Train =", train_scores[i], "Test =", test_scores[i])
# plot the training and testing scores in a log scale
plt.semilogx(param_values, train_scores, label='Train', alpha=0.4, lw=2, c='b')
plt.semilogx(param_values, test_scores, label='X-Val', alpha=0.4, lw=2, c='g')
plt.legend(loc=7)
plt.xlabel(param_name + " values")
plt.ylabel("Mean cross validation accuracy")
# return the training and testing scores on each parameter value
return train_scores, test_scores
In [25]:
alphas = np.logspace(-7, 0, 10)
print(alphas)
In [26]:
train_scores, test_scores = calc_params(X_train, y_train, clf, alphas, 'nb__alpha', 5)
These results suggest that the optimum value of alpha is roughly between 0.03 and 0.05.¶
In [27]:
### Let's use alpha = 0.04 for the final model
mnb = MultinomialNB(alpha=0.04)
In [28]:
mnb.fit(X_tfidf, y_train)
Out[28]:
In [29]:
### Performance on the test data
mnb_pred = mnb.predict(X_test_tfidf)
print(mnb_pred)
In [30]:
from sklearn.metrics import classification_report
print(classification_report(mnb_pred, y_test,
target_names=news.target_names))
Let's now try Support Vector Machines for classification¶
In [31]:
from sklearn.svm import SVC
### Let's again create a pipeline, this time with a linear SVM
clf = Pipeline([
('vect', TfidfVectorizer(
stop_words=stop_words,
token_pattern=r"\b[a-z0-9_\-\.]+[a-z][a-z0-9_\-\.]+\b",
)),
('svc', SVC(kernel='linear')),
])
In [32]:
evaluate_cross_validation(clf, X_train, y_train, 5)
We can try different values of regulariztion parameter C for a better fit¶
In [33]:
c_vals = [1, 5, 10, 50, 100]
train_scores, test_scores = calc_params(X_train, y_train, clf, c_vals, 'svc__C', 5)
It seems that we can obtain a slight improvement with a higher (than the default 1.0) value of C, but not much change occurs beyond C = 10.0)¶
Let's try to do a more systemtic grid search, this time using a Radial Basis Funtion (RBF) kernel. We'll explore the parameter space for teh same values of C as above, as well as for the values of the gamma parameter used to control the width of the RBF function (this is generally a very small value).¶
In [35]:
from sklearn.model_selection import GridSearchCV
parameters = {
'svc__gamma': np.logspace(-3, 0, 4),
'svc__C': [1, 5, 10, 50, 100],
}
clf = Pipeline([
('vect', TfidfVectorizer(
stop_words=stop_words,
token_pattern=r"\b[a-z0-9_\-\.]+[a-z][a-z0-9_\-\.]+\b",
)),
('svc', SVC(kernel='rbf')),
])
gs = GridSearchCV(clf, parameters, verbose=2, cv=3)
In [36]:
%time _ = gs.fit(X, y)
gs.best_params_, gs.best_score_
Out[36]:
With the grid search we obtained a better combination of C and gamma parameters, with values 100.0 and 0.01 respectively, and we obtained a 3-fold cross validation accuracy of 0.829, better than the accuracy obtained (0.807) wiht the linear kernel and the default value of C (1.0).¶
Let's train the final model with these parameters on the full training data and evaluate it on the test set.¶
In [37]:
clf = Pipeline([
('vect', TfidfVectorizer(
stop_words=stop_words,
token_pattern=r"\b[a-z0-9_\-\.]+[a-z][a-z0-9_\-\.]+\b",
)),
('svc', SVC(kernel='rbf', C=100, gamma=0.01)),
])
In [38]:
clf.fit(X_train, y_train)
Out[38]:
In [39]:
svc_pred = clf.predict(X_test)
In [40]:
print(clf.score(X_test, y_test))
In [41]:
print(classification_report(svc_pred, y_test, target_names=news.target_names))
In [42]:
import matplotlib.pyplot as plt
import seaborn as sns; sns.set()
from sklearn.metrics import confusion_matrix
mat = confusion_matrix(y_test, svc_pred)
fig, ax = plt.subplots(figsize=(8,8))
ax = sns.heatmap(mat.T, square=True, linecolor='grey', linewidths=1, annot=True,
fmt='d', cbar=True, cmap='Reds', ax=ax, annot_kws={"fontsize":12, "weight":"bold"},
xticklabels=news.target_names,
yticklabels=news.target_names)
bottom, top = ax.get_ylim()
ax.set_ylim(bottom + 0.5, top - 0.5)
plt.xlabel('true label')
plt.ylabel('predicted label');
In [ ]: