This notebook covers the basics of starting with text data and implementing machine learning algorithms that take that data as input and output predictions.¶

Using scikitlearn, we will walk through:¶

• Naive Bayes Classifier
• Logistic Regression

Naive Bayes Classifier¶

Generative model: Naive Bayes models the joint distribution of the feature X and target Y, and then predicts the posterior probability given as P(y|x)

The naive bayes classifier uses Baye's rule to assign labels to input data, with the output as the label with the highest probability for a number of input features.

Steps:¶

• Loading dataset
• Visualization
• Preprocessing
• Model creation
• Evaluation

Loading the dataset¶

Visualization¶

Scikitlearn Count Vectorizer for Preprocessing¶

Convert a collection of text documents to a matrix of token counts

This implementation produces a sparse representation of the counts using scipy.sparse.csr_matrix.

Steps:¶

• Normalize to lower case
• Remove punctuation
• Tokenize
• Stop word removal
• Bag-of-words representation

Preprocessing the dataset¶

Now lets apply the count vectorizer to our dataset as part of our pre-processing

• Split the dataset into train and test subsets
• Apply count vectorizer to both subset

Creating the model - Naive Bayes Classifier¶

Note that there are different Naive Bayes algorithms available in sklearn. Another commonly used one is Gaussian Naive Bayes which is useful for continuous attributes.

Accuracy: the ratio of correctly predicted observation to the total observations

$$\text{Accuracy}=\frac{\text{TP+TN}}{\text{TP+FP+FN+TN}}$$

Precision: the ratio of correctly predicted positive observations to the total predicted positive observations

$$\text{Precision}=\frac{\text{TP}}{\text{TP+FP}}$$

Recall: the ratio of correctly predicted positive observations to the all observations in actual class

$$\text{Recall}=\frac{\text{TP}}{\text{TP+FN}}$$

F1: the weighted average of Precision and Recall

• takes both false positives and false negatives into account
• useful for uneven class distribution

$$\text{F1}=\frac{\text{2*(Recall * Precision)}}{\text{(Recall + Precision)}}$$

Accuracy is used when the True Positives and True negatives are more important¶

F1-score is used when the False Negatives and False Positives are more important¶

A confusion matrix is useful for visualizing your classifier.¶

• true positives (TP): These are cases in which we predicted yes (they have the disease), and they do have the disease.

• true negatives (TN): We predicted no, and they don't have the disease.

• false positives (FP): We predicted yes, but they don't actually have the disease. (Also known as a "Type I error.")

• false negatives (FN): We predicted no, but they actually do have the disease. (Also known as a "Type II error.")

Logistic Regression¶

Discriminative model: Logistic regression directly models the posterior probability of P(y|x) by learning the input to output mapping by minimising the error.

Uses logit (logistic unit): y = e^(b0 + b1x) / (1 + e^(b0 + b1x)) Used to model binary output variables

• Loading dataset
• Visualization
• Preprocessing
• Model creation
• Evaluation

Loading the dataset¶

Text preprocessing¶

• convert words to vectors
• use TF-IDF to determine word importance

TF-IDF is the product of the TF and IDF scores of the term.

$$\text{TF-IDF}=\frac{\text{TF}}{\text{IDF}}$$

Term Frequency : This summarizes how often a given word appears within a document.

$$\text{TF} = \frac{\text{Number of times the term appears in the doc}}{\text{Total number of words in the doc}}$$

Inverse Document Frequency: This downscales words that appear a lot across documents. A term has a high IDF score if it appears in a few documents. Conversely, if the term is very common among documents (i.e., “the”, “a”, “is”), the term would have a low IDF score.

$$\text{IDF} = \ln\left(\frac{\text{Number of docs}}{\text{Number docs the term appears in}} \right)$$

n-gram is a contiguous sequence of n items from a given sample of text

unigrams are single words

bigram are double words

A metric that you might use for feature selection is chi-squared¶

• The chi-square test measures dependence between stochastic variables
• A "goodness of fit" statistic, because it measures how well the observed distribution of data fits with the distribution that is expected if the variables are independent

Before we can train and test a model, the data needs to be split up into train and test subsets. Scikitlearn's train_test_split package is useful for quickly completing this step - it also has parameters that you can alter to suit your data.

• test_size
• random_state