CMSE 802 Project¶

The Behavioral Analysis of Predictive Modeling on a Bank Marketing Dataset.¶

Author: Jamell Dacon¶

Abstract¶

    I propose a data mining supervised learning approach to predict the accuracy of several models of the client subscription of a term deposit to a bank via information collected via a telephone call; without pruning the data collected from UCI Machine Learning Repository. The largest dataframe was analyzed with 20 predictor variables and a target variable comprising of 41188 examples. The individual comparison of 4 models i.e. Gradient Boosting (GB), Random Forest (RF), Multi-Layer Perceptron (MLP) and Logistic Regression (LGR) models were made along with an 8 model cross-validation comparison comprising of Linear Discriminant Analysis (LDA), K-Nearest Neighbors (KNN), Decision Trees (DT), Support Vector Machine (SVM), Naive Bayes (NB) including the individual model comparison models. Using three metrics such as classification report, accuracy score and a confusion matrix. Both the GB and MLP classifer models produced the best results with an accuracy score of 91.90% and 90.97% respectively, tested on 40% of the data. In the 8 model comparison 10-fold cross validation was implemented were only two models, namely the Gradient Boosting (GB) and Linear Discriminant Analysis (LDA) models produced the best mean accuracy results accomplishing over 90%, whereas the mean of the RF model's accuracy decreased by 1.43%.

Key words: Supervised, Accuracy score, Cross-validation

1. Introduction¶

      Banks are a cruicial part of our economy and may be considered the safest way of saving money over a long-period of time. The commercializing of banks is ubiquitious, meaning it can be found everywhere. One of the many ways to attract customers is by initializing a relationship with a customer with the goal of that customer becoming a client of the bank. This relationship is created by a 1-1 telephone call between the telemarketer and the potential client. With the information received from each potential customer, a data mining approach will be undertaken to gain insight from the information retrieved i.e the available examples to have knowledge of the prediction of a subscription of a client to the bank using classification and regression models. There are several supervised models used such as Gradient Boosting (GB), Random Forest (RF), Multi-Layer Perceptron (MLP), Logistic Regression (LGR) Linear Discriminant Analysis (LDA), K-Nearest Neighbors (KNN), Decision Trees (DT), Support Vector Machine (SVM) and Naive Bayes (NB).

      In this paper, I propose analytical predictive modeling insight of the behavior of several algorithm on predicting the accuracy result of a phone call to a client from a bank telemarketer attemptiong to sell long term deposits by using supervised learning. The main contributions of this work are:
    • A focus on Data Exploration i.e Univariate Analysis was conducted on the largest dataset consisting of 41188 examples with 20 attributes. In the modeling phase, the feature distribution was plotted to gain insight of the visualization of each feature. Futhermore, as over 50% of the data was categorical, encoding allowed the data to be fully numerical which was then passed into models.
    • Several individual model comparisons on a 60/40 split. 
    • A comparison of eight supervised models (GB, RF, LDA, KNN, LGR, SVM, DT and NB). 

2. Methods and Materials¶

2.1. Bank telemarketing dataset information¶

There are four datasets: http://archive.ics.uci.edu/ml/datasets/Bank+Marketing

• bank-additional-full.csv with all examples (41188) and 20 inputs, ordered by date (from May 2008 to November 2010)

• bank-additional.csv with 10% of the examples (4119), randomly selected from bank-additional-full.csv, and 20 inputs.

• bank-full.csv with all examples and 17 inputs, ordered by date (older version of this dataset with less inputs).

• bank.csv with 10% of the examples and 17 inputs, randomly selected from bank-full.csv (older version of this dataset with less inputs).

The smallest datasets are provided to test more computationally demanding machine learning algorithms (e.g., SVM). However, I proposed to carry about this project using the bank-additional-full.csv (largest dataset) since the goal of this project is the efficiently and accurately predict the output using each supervised method to attain knowledge of the robustness of each algorithm.

2.2. Data mining tools¶

2.3. Bank telemarketing data: Attribute Information¶

Input variables:

i. Bank client data:

1. $\underline{age}$ (numeric)
2. $\underline{job}$: type of job (categorical): 'admin.','bluecollar','entrepreneur','housemaid','management','retired','selfemployed','services','student','technician','unemployed','unknown')
3. $\underline{marital}$: marital status (categorical: 'divorced','married','single','unknown'; note: 'divorced' means divorced or widowed)
4. $\underline{education}$ (catgorical): 'basic.4y','basic.6y','basic.9y','high.school','illiterate','professional.course','university.degree','unknown')
5. $\underline{default}$: has credit in default? (categorical: 'no','yes','unknown')
6. $\underline{housing}$: has housing loan? (categorical: 'no','yes','unknown')
7. $\underline{loan}$: has personal loan? (categorical: 'no','yes','unknown')

ii. Related with the last contact of the current campaign: 8. $\underline{contact}$: contact communication type (categorical: 'cellular','telephone') 9. $\underline{month}$: last contact month of year (categorical: 'jan', 'feb', 'mar', ..., 'nov', 'dec') 10. $\underline{day \ of \ week}$: last contact day of the week (categorical: 'mon','tue','wed','thu','fri') 11. $\underline{duration}$: last contact duration, in seconds (numeric).

Important note: this attribute highly affects the output target (e.g., if duration=0 then y='no'). Yet, the duration is not known before a call is performed. Also, after the end of the call y is obviously known. Thus, this input should only be included for benchmark purposes and should be discarded if the intention is to have a realistic predictive model.

iii. Other attributes: 12. $\underline{campaign}$: number of contacts performed during this campaign and for this client (numeric, includes last contact) 13. $\underline{pdays}$: number of days that passed by after the client was last contacted from a previous campaign (numeric; 999 means client was not previously contacted) 14. $\underline{previous}$: number of contacts performed before this campaign and for this client (numeric) 15. $\underline{poutcome}$: outcome of the previous marketing campaign (categorical: 'failure','nonexistent','success')

iv. Social and economic context attributes 16. $\underline{emp.var.rate}$: employment variation rate - quarterly indicator (numeric) 17. $\underline{cons.price.idx}$: consumer price index - monthly indicator (numeric) 18. $\underline{cons.conf.idx}$: consumer confidence index - monthly indicator (numeric) 19. $\underline{euribor3m}$: euribor 3 month rate - daily indicator (numeric) 20. $\underline{nr.employed}$: number of employees - quarterly indicator (numeric)

Output variable (desired target): 21. $\underline{y}$ - has the client subscribed a term deposit? (binary: 'yes','no')

2.5. Data Exploration - Univariate Analysis¶

When exploring our dataset and its features, we have many options available to us. Here we will explore each feature individually, or compare pairs of features, finding the correlation between multiple features is called Multivariate analysis.

Here we explore some simple Univariate (a single feature) analysis. Features can be of multiple types:

• Nominal: is for mutual exclusive, but not ordered, categories.
• Ordinal: is one where the order matters but not the difference between values.
• Interval: is a measurement where the difference between two values is meaningful.
• Ratio: has all the properties of an interval variable, and also has a clear definition of 0.0.

There are multiple ways of manipulating each feature type, but for simplicity, we'll define only two feature types i.e. attributes of features:

• Numerical: any feature that contains numeric values.
  • Nominal
  • Ordinal
• Categorical: any feature that contains categories, or text.
  • Interval
  • Ratio

2.5.1 Univariate Analysis of the Bank Additional Full dataset¶

Now, we have an idea of the attributes types of each column, hence we can then do some analysis on the numerical and categorical columns of the df, along with some visualization those columns.

2.5.2 Feature distribution i.e. visualization of each feature¶

2.5.3 Model Preparation: Encoding¶

Let's check to see that it worked (with the exception of the yHat dummy variable as it should be the same as y encoded):

2.5.4 Explanatory Knowledge: Feature Correlation and Covariance¶

Here we compare pairs of features, by finding the correlation between multiple features, depicted with a heat map for each for the Bank Additional Full dataset.

Question: Why do we want correlation instead of the covariance?

Answer: Covariance and Correlation are two significantly used terms in the field of statistics and probability theory, where we can understand the difference between correlation and covariance matrices, and understand the application of the two in the field of analytics and datasets.

As we see from the formula of covariance (biased),

it assumes the units from the product of the units of the two variables. This is because it divides the value of covariance by the product of standard deviations which have the same units. The value of covariance is affected by the change in scale of the variables. If all the values of the given variable are multiplied by a constant and all the values of another variable are multiplied, by a similar or different constant, then the value of covariance also changes.

On the other hand, correlation is dimensionless i.e. a unit-free measure of the relationship between variables (which is what we need). However, the value of correlation is not influenced by the change in scale of the values.

Lastly, another difference between covariance and correlation is the range of values that they can assume. While correlation coefficients lie between -1 and +1, covariance can take any value between $-\infty$ and $+\infty$

2.6 Feature Selection¶

By the correlation matrix we see that the columns ['emp.var.rate', 'cons.price.idx', 'euribor3m', 'nr.employed'] are heavily correlated as seen by the color of their respective cells. Therefore, they can be dropped leaving features which have a correlation value less than or equal 0.28.

Let's view the new correlation matrix.

3. Experiments¶

3.1 Predictive Modeling¶

The task of predicting the value of an attribute by based on the value of other attributes:

• The attributes that are used to make the prediction are known as the explanatory attributes (also known as independent variables or predictors)
• The attribute to be predicted is known as the taraget attribute (also known as dependent variable, respond variable, or predictand)

3.2 Gradient Boosting Classifier model¶

In many supervised learning problems one has to output the variable $y$, using a training set ${\{(x_1, y_1), ... (x_n, y_n)}\}$ of known values for $x$ corresponding to values of $y$, the goal is to find an approximation $\hat{F}(x)$ to a function $F(x)$ that minimizes the expected value of some specified lost function $L(y, F(x))$:

Gradient boosting is an emsemble machine learning technique that builds the model in a stage-wise fashion as it allows optimization of an arbitrary differentiable loss function.

3.3 Random Forest Classifier model¶

Random decision forests are another ensemble learning method for classification and regression by constructing a multitude of decision trees at training time and outputting the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees.

3.4 MLP Classifier model¶

A multilayer perceptron (MLP) is a class of feedforward artificial neural network. An MLP consists of at least three layers of nodes: an input layer, a hidden layer and an output layer. MLP utilizes a supervised learning technique called backpropagation for training. Its multiple layers and non-linear activation distinguish MLP from a linear perceptron, which can distinguish data that is not linearly separable.

3.5 Logistic Regression model¶

Logistic regression is the appropriate regression analysis to conduct when the dependent variable is dichotomous (binary). Like all regression analyses, the logistic regression is a predictive analysis. Logistic regression is used to describe data and to explain the relationship between one dependent binary variable and one or more nominal, ordinal, interval or ratio-level independent variables.

4. Results¶

4.1 Model Comparison¶

In this subsection, 10 fold cross vaildation is being explored, as the dataset is large each model will have a different behavioral pattern result on each fold, which we will now examine to avoid overfitting and/ or underfitting. Overfitting is often a result of an excessively complicated model, and it can be prevented by fitting multiple models and using validation or cross-validation to compare their predictive accuracies on test data. Intuitively, underfitting occurs when the model or the algorithm does not fit the data well enough. Underfitting is often a result of an excessively simple model, hence we will choose several models and report their mean accuracy and standard deviation.

Cross-validation is a statistical method used to estimate the skill of machine learning models. It is commonly used in applied machine learning to compare and select a model for a given predictive modeling problem because it is easy to understand, easy to implement, and results in skill estimates that generally have lower biases or a less optimistic estimate of the model skill than other methods, such as a simple train/test split.

4.2 Explanatory Knowledge: Model Information¶

Here we can provide some information on the models used.

4.3 Model Behavior¶

The Accuracy table below shows the results of the Model Comparison consisting of the Mean Accuracy and Standard Deviation of each model i.e. the square root of the average of the squared deviations from the mean.

Table 1: Accuracy¶

We can view the results of each fold of each model as named in the Accuracy table above respectively, in which we will create a Model CV results table.

Table 2: Accuracy after Cross-Validation (%)¶

The accuracy percentages in bold represent the best and worst accuracies of the 10-fold cross validation of each model respectively.

4.4 Visualization: Model comparison¶

We now can view the performance of the model comparison with the lines on the exterior of the boxplot represent the highest and lowest accuracies respectively.

5. Conclusion¶

    In previous works a dataset of a massive 150 features related with bank client, product and social-economic attributes were collected, however the data used in this project consisted of 41188 examples with 20 explanatory variables (predictor variables) with very little correlation, a target variable It can be said that alot of the original explanatory variables were previously removed for several reasons which may be high correlation, noise, and/ or useless in the prediction of the target variable.

    In this work, four highly correlated features were removed, further reducing the number of predictor variables as inputs to each model. This can be seen in the second correlation matrix as the highest correlated value is 0.28 i.e a tad larger that 25% pairwise correlation. When the models were trained on an 60/40 split the purpose of the individual model were to test the performance of each model on 40% of the data being trained on 60% of the data, namely 16476 examples where 14655 are 0's and 1821 are 1's. The classification report shows a representation of the main classification metrics on a per-class basis. This gives a deeper intuition of the classifier behavior over global accuracy which can mask functional weaknesses in one class of a multiclass problem. The metrics are defined in terms of true and false positives, and true and false negatives. Positive and negative in this case are generic names for the classes of a binary classification problem. As seen, the precision of the predicition of 1's were typically around 65%, this is a result of an inbalanced dataset since the number of 0's were 8 times as much as 1's. The Neural Network, i.e. Multi-Layer Perceptron consisted of 15 layers, thus the number of weights became increasing large as it was evident in the time complexity of the algorithm despite producing an accuracy score of 90.97%. Although, the Logistic Regression model produced the lowest accuracy score on the data, also with poor precision on prediciting 1's as it generated the largest number of true positives, compared to the other models.

    Cross-validation is primarily used in applied machine learning to estimate the skill of a machine learning model on unseen data. In that, the 10-fold cross validation was applied where several machine learning algorithms were included. The models' behavior varied on each fold, as expected since each fold pertained to a new test set as seen in Table 2. The SVM model was computationally heavy as the input was over 41K examples and resulted in 88.72% accuracy mean. SVM did not perform well on the last fold, hence the decreasing of its accuracy, since employing the kernel trick does not scale well to a substantial amount of training samples, as it takes the form of a function that maps a single vector to a vector of higher dimensionality, thus approximating the kernel. Future work includes carrying out similar supervised ML techniques i.e the precision of 1's are much better. This can be done in several ways as two approaches to this task is undersampling the data to have a balanced dataset, and using the dataset available i.e the bank-additional.csv with 10% of the examples (4119), randomly selected from bank-additional-full.csv. 

6. References¶

• [Moro et al., 2014] S. Moro, P. Cortez and P. Rita. A Data-Driven Approach to Predict the Success of Bank Telemarketing. Decision Support Systems, Elsevier, 62:22-31, June 2014
• http://archive.ics.uci.edu/ml/datasets/Bank+Marketing
• https://scikit-learn.org/stable/supervised_learning.html#supervised-learning
• https://en.wikipedia.org/wiki/Gradient_boosting
• https://en.wikipedia.org/wiki/Random_forest
• https://en.wikipedia.org/wiki/Multilayer_perceptron
• https://www.statisticssolutions.com/what-is-logistic-regression/
• https://en.wikipedia.org/wiki/Decision_tree_learning
• https://scikit-learn.org/stable/modules/neighbors.html
• https://scikit-learn.org/stable/modules/lda_qda.html
• https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
• https://scikit-learn.org/stable/modules/svm.html#svc
• https://scikit-learn.org/stable/modules/naive_bayes.html
• https://www.scikit-yb.org/en/latest/api/classifier/classification_report.html