Introduction: For the validation of any model adapted from a paper, it is of utmost importance that the results of benchmark testing on the datasets listed in the paper match between the actual implementation (palmetto) and gensim. This coherence pipeline has been implemented from the work done by Roeder et al. The paper can be found here.
Approach :
import re
import os
from scipy.stats import pearsonr
from datetime import datetime
from gensim.models import CoherenceModel
from gensim.corpora.dictionary import Dictionary
%load_ext line_profiler # This was used for finding out which line was taking maximum time for indirect confirmation measure
The line_profiler extension is already loaded. To reload it, use: %reload_ext line_profiler
Download the dataset from the link and plug in the location here
prefix = "/home/devashish/datasets/Movies/movie/"
start = datetime.now()
texts = []
for fil in os.listdir(prefix):
for line in open(prefix + fil):
# lower case all words
lowered = line.lower()
#remove punctuation and split into seperate words
words = re.findall(r'\w+', lowered, flags = re.UNICODE | re.LOCALE)
texts.append(words)
end = datetime.now()
print "Time taken: %s" % (end - start)
Time taken: 0:10:23.956500
start = datetime.now()
dictionary = Dictionary(texts)
corpus = [dictionary.doc2bow(text) for text in texts]
end = datetime.now()
print "Time taken: %s" % (end - start)
Time taken: 0:01:44.047829
According to the paper the number of documents should be 108952 with a vocabulary of 1625124. The difference is because of a difference in preprocessing. However the results obtained are still very similar.
print len(corpus)
print dictionary
124234 Dictionary(758123 unique tokens: [u'schelberger', u'mdbg', u'shatzky', u'bhetan', u'verplank']...)
topics = [] # list of 100 topics
for l in open('/home/devashish/datasets/Movies/topicsMovie.txt'):
topics.append([l.split()])
topics.pop(100)
[[]]
human_scores = []
for l in open('/home/devashish/datasets/Movies/goldMovie.txt'):
human_scores.append(float(l.strip()))
start = datetime.now()
u_mass = []
flags = []
for n, topic in enumerate(topics):
try:
cm = CoherenceModel(topics=topic, corpus=corpus, dictionary=dictionary, coherence='u_mass')
u_mass.append(cm.get_coherence())
except KeyError:
flags.append(n)
end = datetime.now()
print "Time taken: %s" % (end - start)
Time taken: 0:20:44.833342
This is expected to take much more time since c_v
uses a sliding window to perform probability estimation and uses the cosine similarity indirect confirmation measure.
start = datetime.now()
c_v = []
for n, topic in enumerate(topics):
try:
cm = CoherenceModel(topics=topic, texts=texts, dictionary=dictionary, coherence='c_v')
c_v.append(cm.get_coherence())
except KeyError:
pass
end = datetime.now()
print "Time taken: %s" % (end - start)
Time taken: 19:50:11.214341
They should be taking lesser time than c_v but should have a higher correlation than u_mass
start = datetime.now()
c_uci = []
flags = []
for n, topic in enumerate(topics):
try:
cm = CoherenceModel(topics=topic, texts=texts, dictionary=dictionary, coherence='c_uci')
c_uci.append(cm.get_coherence())
except KeyError:
flags.append(n)
end = datetime.now()
print "Time taken: %s" % (end - start)
Time taken: 2:55:36.044760
start = datetime.now()
c_npmi = []
for n, topic in enumerate(topics):
print n
try:
cm = CoherenceModel(topics=topic, texts=texts, dictionary=dictionary, coherence='c_npmi')
c_npmi.append(cm.get_coherence())
except KeyError:
pass
end = datetime.now()
print "Time taken: %s" % (end - start)
Time taken: 2:53:55.424213
final_scores = []
for n, score in enumerate(human_scores):
if n not in flags:
final_scores.append(score)
One topic encountered a KeyError. This was because of a difference in preprocessing due to which one topic word wasn't found in the dictionary
print len(u_mass), len(c_v), len(c_uci), len(c_npmi), len(final_scores)
# 1 topic has word(s) that is not in the dictionary. Probably some difference
# in preprocessing
99 99 99 99 99
The values in the paper were:
u_mass
correlation : 0.093
c_v
correlation : 0.548
c_uci
correlation : 0.473
c_npmi
correlation : 0.438
Our values are also very similar to these values which is good. This validates the correctness of our pipeline.
print pearsonr(u_mass, final_scores)[0]
print pearsonr(c_v, final_scores)[0]
print pearsonr(c_uci, final_scores)[0]
print pearsonr(c_npmi, final_scores)[0]
0.133916622716 0.555948711374 0.414722858726 0.39935634517