Ensemble Classifiers: Random Forest and ADA Boost¶
Two families of ensemble methods are usually distinguished:¶
In averaging methodss, the driving principle is to build several estimators independently and then to average their predictions. On average, the combined estimator is usually better than any of the single base estimator because its variance is reduced. Examples: Bagging methods, Forests of randomized trees, …
By contrast, in boosting methods, base estimators are built sequentially and one tries to reduce the bias of the combined estimator. The motivation is to combine several weak models to produce a powerful ensemble. Examples: AdaBoost, Gradient Tree Boosting, …
In [1]:
%pylab inline
In [2]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
Import titanic data using pandas¶
Modified version of the "Titanic" data can be found at: http://http://facweb.cs.depaul.edu/mobasher/classes/csc478/Data/titanic-trimmed.csv. Original unmodified Titanic data is available at CMU StatLib.
In [3]:
url = "http://facweb.cs.depaul.edu/mobasher/classes/csc478/Data/titanic-trimmed.csv"
titanic = pd.read_csv(url)
titanic.head(10)
Out[3]:
In [4]:
titanic.describe(include="all")
Out[4]:
Handling missing variables¶
In [5]:
titanic[titanic.age.isnull()].shape
Out[5]:
In [6]:
age_mean = titanic.age.mean()
titanic.age.fillna(age_mean, axis=0, inplace=True)
titanic.dropna(axis=0, inplace=True)
In [7]:
titanic.shape
Out[7]:
In [8]:
titanic.set_index('pid', drop=True, inplace=True)
titanic.head()
Out[8]:
In [9]:
titanic.describe().T
Out[9]:
In [10]:
titanic_ssf = pd.get_dummies(titanic)
titanic_ssf.head(10)
Out[10]:
In [11]:
titanic_names = titanic_ssf.columns.values
titanic_names
Out[11]:
In [12]:
y = titanic_ssf['survived']
X = titanic_ssf[titanic_names[1:]]
X.head()
Out[12]:
Build the training and testing dataset¶
In [13]:
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=33)
A versatile function to measure performance of a model¶
In [14]:
from sklearn import metrics
def measure_performance(X, y, clf, show_accuracy=True, show_classification_report=True, show_confussion_matrix=True):
y_pred = clf.predict(X)
if show_accuracy:
print("Accuracy:{0:.3f}".format(metrics.accuracy_score(y, y_pred)),"\n")
if show_classification_report:
print("Classification report")
print(metrics.classification_report(y, y_pred),"\n")
if show_confussion_matrix:
print("Confussion matrix")
print(metrics.confusion_matrix(y, y_pred),"\n")
Let's now compare performance of standard decision tree classifier against some ensemble methods: Random Forest Classifier and Ada Boost.¶
In [15]:
from sklearn import tree
dt = tree.DecisionTreeClassifier(criterion='gini')
dt = dt.fit(X_train, y_train)
In [16]:
from sklearn import metrics
measure_performance(X_test, y_test, dt, show_confussion_matrix=False, show_classification_report=False)
In [17]:
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators=10, random_state=33)
rf = rf.fit(X_train, y_train)
In [18]:
measure_performance(X_test, y_test, rf, show_confussion_matrix=False, show_classification_report=False)
Exploring and comparing model parameters¶
In [19]:
print(rf.get_params())
Let's use our "calc_params" function to explore the impact of individual parameters using cross-validation¶
In [20]:
from sklearn.model_selection import KFold
def calc_params(X, y, clf, param_values, param_name, K):
# Convert input to Numpy arrays
X = np.array(X)
y = np.array(y)
# initialize training and testing score arrays with zeros
train_scores = np.zeros(len(param_values))
test_scores = np.zeros(len(param_values))
# iterate over the different parameter values
for i, param_value in enumerate(param_values):
# set classifier parameters
clf.set_params(**{param_name:param_value})
# initialize the K scores obtained for each fold
k_train_scores = np.zeros(K)
k_test_scores = np.zeros(K)
# create KFold cross validation
cv = KFold(n_splits=K, shuffle=True, random_state=0)
# iterate over the K folds
j = 0
for train, test in cv.split(X):
# fit the classifier in the corresponding fold
# and obtain the corresponding accuracy scores on train and test sets
clf.fit(X[train], y[train])
k_train_scores[j] = clf.score(X[train], y[train])
k_test_scores[j] = clf.score(X[test], y[test])
j += 1
# store the mean of the K fold scores
train_scores[i] = np.mean(k_train_scores)
test_scores[i] = np.mean(k_test_scores)
print(param_name, '=', param_value, "Train =", train_scores[i], "Test =", test_scores[i])
# plot the training and testing scores in a log scale
plt.plot(param_values, train_scores, label='Train', alpha=0.4, lw=2, c='b')
plt.plot(param_values, test_scores, label='X-Val', alpha=0.4, lw=2, c='g')
plt.legend(loc=7)
plt.xlabel(param_name + " values")
plt.ylabel("Mean cross validation accuracy")
# return the training and testing scores on each parameter value
return train_scores, test_scores
Now we can explore the impact of 'min_samples_leaf' more systematically¶
In [21]:
msl = range(1,6)
print(msl)
In [22]:
rf = RandomForestClassifier(n_estimators=10, random_state=33)
train_scores, test_scores = calc_params(X_train, y_train, rf, msl, 'min_samples_leaf', 5)
Seems that min_sample_leaf = 3 works well in this case. Let's now try varying tree max depths¶
In [23]:
m_depth = [1,2,3,4,5,6, 7, 8]
rf = RandomForestClassifier(n_estimators=10, random_state=33)
train_scores, test_scores = calc_params(X_train, y_train, rf, m_depth, 'max_depth', 5)
Looks like the best max_depth values is 4. Let's now try varying the number of estimators in the ensemble¶
In [24]:
nest = range(5, 101, 5)
print(nest)
In [25]:
rf = RandomForestClassifier(n_estimators=10, random_state=33)
train_scores, test_scores = calc_params(X_train, y_train, rf, nest, 'n_estimators', 5)
Not a bif difference, but 25 works well.. Let's build and test our final model with these parameters¶
In [26]:
rf = RandomForestClassifier(n_estimators=25, min_samples_leaf=3, max_depth=4)
rf = rf.fit(X_train, y_train)
measure_performance(X_test, y_test, rf, show_confussion_matrix=False, show_classification_report=False)
We can also observe the importance of each feature¶
In [27]:
rf.feature_importances_
Out[27]:
In [28]:
def plot_feature_importances(model, n_features, feature_names):
plt.barh(range(n_features), model.feature_importances_, align='center')
plt.yticks(np.arange(n_features), feature_names)
plt.xlabel("Feature importance")
plt.ylabel("Feature")
plt.ylim(-1, n_features)
features = titanic_names[1:]
plot_feature_importances(rf, len(features), features)
And we can access each of the 25 estimators¶
In [29]:
rf.estimators_[:3]
Out[29]:
In [30]:
fig, ax = plt.subplots(figsize=(50,20))
tree.plot_tree(rf.estimators_[0], feature_names=features, class_names=["No","Yes"], filled=True, ax=ax);
In [31]:
import graphviz
from sklearn.tree import export_graphviz
dot_data = export_graphviz(rf.estimators_[0],out_file=None, feature_names=features,
class_names=["No","Yes"], filled=True, rotate=True)
graph = graphviz.Source(dot_data)
graph
#import pydotplus
#from sklearn.externals.six import StringIO
#from IPython.display import Image
#dot_data = StringIO()
#export_graphviz(rf.estimators_[0],out_file=dot_data, feature_names=features, class_names=["Yes","No"])
#graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
#Image(graph.create_png())
Out[31]:
Ada Boost¶
In [32]:
from sklearn.ensemble import AdaBoostClassifier
In [33]:
ab = AdaBoostClassifier()
ab = ab.fit(X_train, y_train)
In [34]:
measure_performance(X_test, y_test, ab, show_confussion_matrix=False, show_classification_report=False)
In [35]:
train_scores, test_scores = calc_params(X_train, y_train, ab, nest, 'n_estimators', 5)
In [36]:
ab = AdaBoostClassifier(n_estimators=10)
ab = ab.fit(X_train, y_train)
measure_performance(X_test, y_test, ab, show_confussion_matrix=False, show_classification_report=False)
Learning rate shrinks the contribution of each classifier by "learning_rate". There is a trade-off between "learning_rate" and "n_estimators"¶
In [37]:
lr = [0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.6, 0.7, 0.8, 0.9, 1.0, 1.3, 1.5, 1.8, 2.0]
In [38]:
ab = AdaBoostClassifier()
train_scores, test_scores = calc_params(X_train, y_train, ab, lr, 'learning_rate', 5)
In [39]:
ab = AdaBoostClassifier(learning_rate=1.3)
ab = ab.fit(X_train, y_train)
measure_performance(X_test, y_test, ab, show_confussion_matrix=False, show_classification_report=False)
Let's use grid search to explore the parameter space more systematically¶
In [40]:
from sklearn.model_selection import GridSearchCV
In [41]:
ab = AdaBoostClassifier()
In [42]:
parameters = {
'learning_rate': [0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.6, 0.7, 0.8, 0.9, 1.0, 1.3, 1.5, 1.8, 2.0],
'n_estimators': range(5, 101, 5),
}
gs = GridSearchCV(ab, parameters, cv=3)
In [43]:
%time _ = gs.fit(X_train, y_train)
In [44]:
gs.best_params_, gs.best_score_
Out[44]:
Our final AdaBoost model;¶
In [45]:
ab = AdaBoostClassifier(n_estimators=20, learning_rate=1.3)
ab = ab.fit(X_train, y_train)
measure_performance(X_test, y_test, ab, show_confussion_matrix=False, show_classification_report=False)
Let's try a couple of other boosting and bagging enssemble classifiers¶
The following is an example of using Gradient Boosted Decision Trees. GBDT is a generalization of boosting to arbitrary differentiable loss functions. GBDT is an accurate and effective procedure that can be used for both regression and classification.¶
In [46]:
from sklearn.ensemble import GradientBoostingClassifier
In [47]:
gb = GradientBoostingClassifier(n_estimators=45, learning_rate=1.0,
max_depth=3, random_state=0).fit(X_train, y_train)
In [48]:
gb.score(X_test, y_test)
Out[48]:
In scikit-learn, bagging methods are offered as a unified BaggingClassifier meta-estimator (resp. BaggingRegressor), taking as input a user-specified base estimator along with parameters specifying the strategy to draw random subsets. For example, the following is a bagging ensemble of KNeighborsClassifier base estimators, each built on random subsets of 50% of the samples and 50% of the features.¶
In [49]:
from sklearn.ensemble import BaggingClassifier
from sklearn.neighbors import KNeighborsClassifier
bagging = BaggingClassifier(KNeighborsClassifier(), max_samples=0.5, max_features=0.5,
n_estimators=25, random_state=5)
bagging.fit(X_train, y_train)
Out[49]:
In [50]:
bagging.score(X_test, y_test)
Out[50]:
In [ ]: