Example of transforming a term-document matrix using TFxIDF (Term Frequency x Inverse Document Frequency)¶
In [1]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
In [2]:
Data = pd.read_csv("http://facweb.cs.depaul.edu/mobasher/classes/csc478/data/term-doc-mat.csv", header=None)
Data
Out[2]:
In [3]:
# Let's remove the column containing the terms
# TD will be out term x document matrix
TD = Data.iloc[:,1:]
TD
Out[3]:
In [4]:
# Reindex the columns to start from 0
TD.columns= range(15)
TD
Out[4]:
In [5]:
# The list of our index terms
terms = Data.iloc[:,0]
terms
Out[5]:
In [6]:
TD.shape
Out[6]:
In [7]:
numTerms=TD.shape[0]
NDocs = TD.shape[1]
In [8]:
print(numTerms)
print(NDocs)
Next, let's compute term frequencies to get an idea of their distributions across the corpus.¶
In [9]:
termFreqs = TD.sum(axis=1)
termFreqs
Out[9]:
In [10]:
plt.plot(sorted(termFreqs, reverse=True))
plt.show()
Next, we will transform the data to TFxIDF weights:¶
In [11]:
# Note: doc frequency (df) for a term t is the number of docs in which t appears divided by total number of docs
# first let's find the doc counts for each term
DF = pd.DataFrame([(TD!=0).sum(1)]).T
DF
Out[11]:
In [12]:
# Create a matrix with all entries = NDocs
NMatrix=np.ones(np.shape(TD), dtype=float)*NDocs
np.set_printoptions(precision=2,suppress=True,linewidth=120)
print(NMatrix)
In [13]:
# Convert each entry into IDF values
# IDF is the log of the inverse of document frequency
# Note that IDF is only a function of the term, so all columns will be identical.
IDF = np.log2(np.divide(NMatrix, np.array(DF)))
In [14]:
print(IDF)
In [15]:
# Finally compute the TFxIDF values for each document-term entry
TD_tfidf = TD * IDF
In [16]:
pd.set_option("display.precision", 2)
TD_tfidf
Out[16]:
Let's now repeat the k-nearest-neighbor document retrieval example from earier notebook, but this time, we'll use the TFxIDF weighted document vectors.¶
In [17]:
def knn_search(x, D, K, measure):
""" find K nearest neighbors of an instance x among the instances in D """
if measure == 0:
# euclidean distances from the other points
dists = np.sqrt(((D - x)**2).sum(axis=1))
elif measure == 1:
# first find the vector norm for each instance in D as wel as the norm for vector x
D_norm = np.array([np.linalg.norm(D[i]) for i in range(len(D))])
x_norm = np.linalg.norm(x)
# Compute Cosine: divide the dot product o x and each instance in D by the product of the two norms
sims = np.dot(D,x)/(D_norm * x_norm)
# The distance measure will be the inverse of Cosine similarity
dists = 1 - sims
idx = np.argsort(dists) # sorting
# return the indexes of K nearest neighbors
return idx[:K], dists
Let's now try this on a new query object as a test instance¶
In [18]:
x = np.array([3, 22, 0, 17, 9, 6, 1, 12, 0, 22])
Need to also perform TFxIDF transformation on the query¶
In [19]:
# Each term in query x must be multiplied by the idf value of the term we computed earlier (the IDF matrix)
x_tfidf = x * IDF.T[0] # note that this coordinatewise multiplication of two vectors
print(x_tfidf)
In [20]:
# The KNN Search function expects a document x term materix as an np array, so we need to transpose the TF_tfidf matrix
DT_tfidf = TD_tfidf.T
DT_array = np.array(DT_tfidf)
In [21]:
# Finding the k=5 nearest neighbors using inverse of Cosine similarity as a distance metric
neigh_idx, distances = knn_search(x_tfidf, DT_array, 5, 1)
In [22]:
distances = pd.Series(distances, index=DT_tfidf.index)
distances.sort_values()
Out[22]:
In [23]:
print("Query:", x)
print("\nNeighbors:")
DT_tfidf.iloc[neigh_idx]
Out[23]:
If you compare this result to the one from the previous notebook (where we did not use TFxIDF), you'll note that document 12 was demoted in the new ranking (it was previously in the 3rd position). This is likely because this document matched strongly with the query on terms 3 and 9. Both of these terms occur frequently across the documents and so the TFxIDF transformation resulted in their weights being penalized.¶
Next, let's extend this example to classification using the KNN approach.¶
In [24]:
# Let's add some labels to our original data
cat_labels = np.array(["Databases", "Databases", "Databases", "Databases", "Databases", "Regression", "Regression", "Regression", "Regression", "Regression", "Information Retrieval", "Information Retrieval", "Information Retrieval", "Information Retrieval", "Information Retrieval"])
cat_labels = pd.Series(cat_labels, index=DT_tfidf.index)
DT_tfidf["Category"] = cat_labels
DT_tfidf
Out[24]:
The function below will use our previous knn_search function to identify the label/category for for an instance x to be classifed, using the majority label of the K nearest neighbors of x.¶
In [25]:
def knn_classify(x, D, K, labels, measure):
from collections import Counter
neigh_idx, distances = knn_search(x, D, K, measure)
neigh_labels = labels[neigh_idx]
count = Counter(neigh_labels)
print("Labels for top ", K, "neighbors: ", count)
return count.most_common(1)[0][0]
In [26]:
print("Instance to classify:\n", x)
print("Predicted Category for the new instance: ", knn_classify(x_tfidf, DT_array, 5, cat_labels, 1))
In [ ]: