TextCL Tutorial¶

1. Introduction
2. Preparation
3. Filtering on language
4. Filtering on Jaccard similarity
5. Filtering on perplexity score
6. Outliers filtering
7. Plots for outlier detection

Introduction¶

This tutorial demonstrates how to preprocess text to clean it up for the modeling stage. Modeling can include classification models, NER (Named Entity Recognition) models, spell checking models, text summarization and generation, as well as prediction models. Usually, this analysis is performed manually and is very time-consuming. The TextCL package helps to identify and filter out (1) sentences in languages other than the target language, (2) linguistically unconnected and/or corrupted sentences, and (3) duplicate sentences.

Another feature of the package is to identify and filter out outliers from the text scope. As outliers, we consider texts that don't contextually belong to the main topic of the text. It's important to be able to identify these anomalies without having labeled data, so we can have a general algorithm for unstructured texts and find out the scope blocks.

In this tutorial, we will work with the BBC data set and additional manually generated sentences to demonstrate the package's functionality. Overall, the package can be used with any text data set loaded as a Pandas data frame.

The package contains the following functions for text cleaning:

1. Filtering on language
2. Filtering on Jaccard similarity
3. Filtering on perplexity score
4. Outliers filtering

The first three functions work at the sentence level for each text in the scope. In turn, the outlier filtering function works at the level of the full text. The latter implements 3 different outlier detection algorithms: TONMF, RPCA, and SVD, with l2 normalisation by default (can be changed to the l1, l2, or max via the norm parameter).

Preparation¶

Load package and dependencies:

Let's prepare the input data from the modified BBC dataset bundled with TextCL. First, load the text data. The sample file is located in the examples folder, next to this tutorial.

In this example we'll use a subset of the BBC News data set containing 5 topics (business, entertainment, politics, sport, tech) with manually inserted texts to demonstrate the capabilities of the package. Here's how the dataset looks like:

Split texts into sentences¶

To be able to process/filter sentences from the data set separately, we first need to split our texts into sentences as rows in a Pandas data frame. The split_into_sentences() function is used for this purpose.

Notice the data loaded in the previous section has a column named text. By default, the split_into_sentences() function expects to find texts in this column. However, an alternative name for this column can be specified in the function's text_col parameter.

Splitting the data set texts into sentences is done as follows, using the split_into_sentences() function:

After splitting text data set into sentences, the number of rows increased from 21 to 319. Let's review them:

As shown in this example, the split_into_sentences() function places sentences in the sentence column by default. This can be changed using the function's sentence_col parameter.

Filtering on language¶

Let's check how the language filtering function works with manually inserted texts in Russian and Turkish languages to the initial set.

To do this we'll use the language_filtering() function, which filters sentences by language. The input to this function should be a Pandas data frame (with sentence column), a threshold value, and a target language. Language score is the threshold used for filtering, with the default value of 0.99. The function makes use of the detect_language function from the langdetect package, which returns probabilities of a text belonging to a certain language. All sentences below this threshold will be filtered out.

The number of rows with sentences was reduced from 319 to 281. Join sentences to the initial texts to review the results:

As we can see, texts with index 3 (Turkish), 7 (Russian) and 8 (Turkish) were removed.

Filtering on Jaccard similarity¶

Function jaccard_sim_filtering() is used to filter sentences by Jaccard similarity. It represents each sentence as an array of tokens and finds the intersection between each pair of arrays / sentences. Using the intersection, the similarity score is calculated; if it's above the specified threshold, a sentence will be filtered out.

The number of rows with sentences was reduced from 281 to 258. Join sentences to the initial texts to review the results:

Text with $id=17$ was removed as it was a partial duplicate of text with $id=18$.

Filtering on perplexity score¶

Function perplexity_filtering() is used to filter sentences by perplexity, i.e., when sentences are linguistically incorrect and/or unconnected with the remaining text.

In general, perplexity is a measurement of how well a probability distribution or probability model predicts a sample. In the case of the text data, we will be checking the probability of the next word to be in the given sentence. A low perplexity indicates the probability distribution is good at predicting the word.

The first step creates contextual tokens to capture latent syntactic-semantic information provided by the pytorch_pretrained_bert package with pretrained openai-gpt tokenizer and using GPT as a language model with OpenAIGPTLMHeadModel. Perplexity is calculated as exp(loss) (where loss is the language modeling loss of a particular token).

The number of rows with sentences was reduced from 258 to the 246. Join sentences to the initial texts to review the results:

Text with $id=19$ was removed because sentence "data clear additional 78.0 long-term 43 those)" is not linguistically correct.

Outliers filtering¶

Text data is uniquely challenging to outlier detection because of its sparsity and high-dimensional nature. In this package we will use Non-Negative Matrix Factorization (NMF) to detect the text topics in an unsupervised fashion, so the model can be trained and be able to detect outliers without labeling of topics.

The way it works is that NMF decomposes high-dimensional vectors into a lower-dimensional representation. These lower-dimensional vectors are non-negative, which also means their coefficients are non-negative. With this approach, we can see the fact that NMF is similar to probabilistic latent semantic indexing (pLSI) and latent Dirichlet allocation (LDA) generative models. From the original matrix, $A$, NMF yields two matrices, $W$ and $H$. The former represents the topics detected in the text data set, while the latter contains the weights for those topics. In other words, $A$ is the documents by words matrix, $H$ is the articles by topics matrix and $W$ is the topics by words matrix.

The outlier_detection() function is used to detect outliers in a list of sentences based on the contextual information using unsupervised methods. Text embeddings are created as a bag of words as an input for the implemented algorithms. The main input parameters for this function are the Pandas data frame containing the texts, the method to use for outlier detection (TONMF by default), and the type of norm (l2 by default) to normalize the obtained matrix and detecting the unrelated texts.

To test this, we'll first join sentences to the text after filtering and selecting the "tech" category. This category has a manually inserted outlier ($id=20$) with a person profile instead of text on the subject of "tech".

Let's run the outlier_detection() function with the RPCA algorithm:

The contents of joined_texts are now as follows:

Text with $id=20$ was removed because it describes a person profile instead of tech news.

Plots for outlier detection¶

In this section we'll visualize the outliers for the BBC data set obtained with the different algorithms. First load additional libraries required for results visualization:

We'll use the NLTK library to filter stop words from the initial dataset. Let's import this library and download the stop words list:

Prepare input data from the BBC dataset¶

To demonstrate how the package handles outlier detection tasks, we present a simplified example from a real world data set to show how skewed the typical values of the corresponding column $z(x)$ may be in real scenarios. To build plots for the outlier detection functions we'll need to load the full BBC dataset and apply initial basic text cleaning:

Data set contains 2225 texts from the following categories:

Get data for "business" and "politics" classes to form the core of the data set:

Get 50 random outliers documents from "tech" class:

Add outliers documents to the core data set and shuffle:

Convert texts to a bag of words using sklearn's CountVectorizer class:

Note that CountVectorizer is used as an example. We could have used another tokenizer or word embedding.

In the following subsections we'll plot the outlier detection results using the three methods included with TextCL, namely TONMF, RPCA, and SVD.

TONMF¶

The tonmf() function uses the TONMF algorithm to obtain the outlier matrix. The solution is based on the non-negative matrix factorization with the extension of the block coordinate descent framework.

Normalize with l2-normalization:

Use ROC curve to plot the results of the algorithm:

Display ℓ2 norm of columns of $Z$ outlier matrix:

RPCA¶

The function rpca_implementation() uses Robust Principal Component Analysis (RPCA) to obtain the outlier matrix. RPCA uses low-rank approximation and yields two matrices: low-rank matrix $L$ and a sparse matrix $S$. After normalization, the $S$ matrix represents the outlier score for the document.

Normalize with l2-normalization:

Display ℓ2 norm of columns of Z outlier matrix:

SVD¶

The function svd() uses singular value decomposition (SVD) to obtain the outlier matrix. SVD is performed with the np.linalg function from the numpy package. The outlier matrix is presented as the multiplication of square root of diagonal elements of the rectangular diagonal matrix $S$ and complex unitary matrix.

Normalize with l2-normalization:

Display ℓ2 norm of columns of Z outlier matrix:

The ROC-curve plot was used for the evaluation of the algorithm results. By analyzing obtained plots in these 3 examples we can see that with the current parameters for the BBC data set the RPCA method is offering the best results. In the plot "Results of RPCA+L2 (outliers - red, non-outliers - blue)" we can see the abnormal texts have higher values of the ℓ2 norm and can be segmented from the main distribution. AUC here represents the degree of separability between the main text group and outlied texts, the higher the AUC, the better the model is at distinguishing between different types of texts in the dataset.