1. pydataberlin-2017
  2. notebook

Evaluating Topic Models

PyData Berlin 2017 Talk

This notebook is a companion to the talk I gave at the PyData Berlin 2017 conference on evaluating topic models

Unsupervised models in natural language processing (NLP) have a long history but have recently become very popular. Word2vec, GloVe, LSI and LDA provide powerful computational tools to deal with natural language and make exploring and modelling large document collections feasible.

Often evaluating the model output requires an existing understanding of what should come out. For topic models the output should reflect our understanding of the relatedness of topical categories, for instance sports, travel or machine learning. Distributional models of language such as word2vec and GloVe should capture some, or ideally all, of the semantics of how language is used.

This is a lot to ask! Not necessarily because it isn't learneable, after all we've learned it, but because we are not necessarily able to represent the desired output as an evaluation function and data set that can be optimised. As an example topic models are often evaluated with respect to the semantic coherence of the topics based on a set of top words from the topics. It is not clear if a set of words such as {cat, dog, horse, pet} captures the semantics of an animalness or a petsiness fully. Nevertheless these methods are useful in determining if the distributed word representation are capturing some of the information conveyed by words and if a topic model is understandable to a human.

This notebook explores a number of these issues in context and aims to provide an overview of the research that has been done in the past 10 or so years, mostly focusing on topic models.

The notebook is split into three parts

  1. Eye Balling models
    • ways of making visual, manual inspection of models easier
  2. Intrinsic Evaluation Methods
    • how to measure the internal coherence of topic models
  3. Putting a Number on Human Judgements
    • quantitative methods for evaluating human judgement

Random collection of other stuff

While preparing the talk and the notebook I experimented with a lot of different software packages and corpora. These are dumped as a somewhat unorganised collection of "other things" at the end of the notebook

Why Evaluate Models

We would like to be able to say if a model is objectively good or bad, and compare different models to each other. This requires us to have an objective measure for the quality of the model but many of the tasks mentioned above require subjective evaluation.

In practical applications one needs to evaluate if "the correct thing" has been learned, often this means applying implicit knowledge and "eye-balling". Documents that talk about football should be in the same category and cat is more similar to dog than to pen. Ideally this information should be captured in a single metric that can be maximised. It is not clear how to formulate such a metric however, over the years there has been numerous attempts from various different angles at formulating semantic coherence, none capture the desired outcome fully and there are numerous issues one should be aware of in applying those metrics.

Some of the issues are related to the metrics being used or issues one should be aware of when applying those metrics, but others are related to external factors, like which kind of held out data to use. Natural language is messy, ambiguous and full of interpretation, that's where a lot of the expressive richness comes from. Sometimes trying to cleanse the ambiguity also reduces language to an unnatural form.

Topic Models

Topic models aim to capture the way in which words co-occur in the context of a document and divide the source corpus into some number of (soft) clusters. There are a large number of variations on the topic model, initial work was done be Deerwester in developing Latent Semantic Analysis (LSA/LSI), now the canonical example is Latent Dirichlet Allocation or LDA. The unit upon which topic models work is a sparse document-term matrix depicted below.

Each row is a document, each column is a term and the entries in each cell usually represent the frequency of each term in each document.

In [127]:
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer

documents = ['The cat sat on the mat', 'A red cat sat on the mat', 'No cat sat on the mat']

vectoriser = CountVectorizer().fit(documents)
X = vectoriser.transform(documents)
pd.DataFrame(X.A, columns=sorted(vectoriser.vocabulary_.keys(), key=lambda k: vectoriser.vocabulary_[k]))
Out[127]:
cat mat no on red sat the
0 1 1 0 1 0 1 2
1 1 1 0 1 1 1 1
2 1 1 1 1 0 1 1

In the case of LDA a Bayesian model is fitted using the data from this matrix. Each topic in the model becomes a probability distribution over terms (the columns). Conceptually this is saying that semantic concepts can be represented as probabilities over a set of words. This makes sense as the topic of discussion acts as a limiting factor on the vocabulary one is likely to use, hear or read in the context of that dicussion.

Words relating to political campaining are much less likely to be observed in documents that discuss ice hockey. Notice however that is is unlikely not impossible, it is not the case that it can not ever happen, it is simply statistically less likely to be the case that caucus or polling will be in a document that otherwise discusses the Teemu Selänne retiring. A topic therefore is a probability distribution over the entire vocabulary, indicating how likely each word is to occur within that topic.

The documents the model is built over can be as short as a single sentence (a tweet) or as long as a chapter in a book. Typically very short documents tend to be more difficult to built coherent models over than slightly longer documents.

Open source implementation of the models are readily available

A Model

In order to evaluate a model, we must of course have one. I'll use the same model(s), built from the Fake News data set on Kaggle, throughout this notebook.

In [4]:
import pandas as pd

df_fake = pd.read_csv('/usr/local/scratch/data/kaggle/fake.csv')
df_fake[['title', 'text', 'language']].head()
Out[4]:
title text language
0 Muslims BUSTED: They Stole Millions In Gov’t B... Print They should pay all the back all the mon... english
1 Re: Why Did Attorney General Loretta Lynch Ple... Why Did Attorney General Loretta Lynch Plead T... english
2 BREAKING: Weiner Cooperating With FBI On Hilla... Red State : \nFox News Sunday reported this mo... english
3 PIN DROP SPEECH BY FATHER OF DAUGHTER Kidnappe... Email Kayla Mueller was a prisoner and torture... english
4 FANTASTIC! TRUMP'S 7 POINT PLAN To Reform Heal... Email HEALTHCARE REFORM TO MAKE AMERICA GREAT ... english
In [5]:
import numpy as np

df_fake = df_fake.loc[(pd.notnull(df_fake.text)) & (df_fake.language == 'english')]
df_fake.shape
Out[5]:
(12357, 20)

There is a total of 12357 non empty english language documents, should be enough to build a model. Let's parse the documents using spacy, getting rid of some non content words and chuck that into gensim. I'll use the gensim.corpora.MmCorpus to serialise the text onto disk, this both saves memory and allows random access to the corpus, which will become useful later for creating different splits of the data.

In [4]:
import spacy

import gensim
from gensim.models import LdaModel
from gensim.corpora import Dictionary, MmCorpus

spc = spacy.load('en')

KEEP_POS = set([90, 98, 82, 84, 94])  # NOUN, VERB, ADJ, ADV, PROPN
pipe = spc.pipe(df_fake.text, parse=False, entity=False, n_threads=8)
processed = [[token.lemma_ for token in document if token.pos in KEEP_POS] for document in pipe]

vocabulary = Dictionary(processed)
vocabulary.filter_extremes(no_below=3, no_above=0.5)
In [133]:
MmCorpus.serialize('./fake_news.mm', (vocabulary.doc2bow(doc) for doc in processed))
In [8]:
vocabulary.save('./fake_news.vocab')
In [ ]:
del processed
In [5]:
corpus_fake = MmCorpus('./fake_news.mm')
In [137]:
lda_fake = LdaModel(corpus=corpus_fake, id2word=vocabulary, num_topics=35, chunksize=1500, iterations=200, alpha='auto')

Inspecting the top 6 six word from the model we can certainly identify some structure. Ther are topic about the Flint Michigan water crisis (Topic 11), the Dakota Access Pipeline (Topic 9) protests and the US elections.

In [7]:
pd.DataFrame([[word for rank, (word, prob) in enumerate(words)]
              for topic_id, words in lda_fake.show_topics(formatted=False, num_words=6, num_topics=35)].iloc[])
Out[7]:
0 1 2 3 4 5
0 dna video abortion veritas project know
1 al syrian force mosul aleppo isis
2 black obama white people soros america
3 china which alien would there ufo
4 get go ’ just know there
5 russia russian putin nato missile nuclear
6 market cancer product bank study report
7 use drug health company report medical
8 post comment facebook news video result
9 pipeline dakota police water rock protester
10 child school police family when people
11 water study health toxic flint tea
12 clinton campaign hillary email podesta wikileaks
13 clinton assange hillary there trump would
14 obama money bank hillary president make
15 party care kelly insurance obamacare year
16 trump vote clinton election hillary voter
17 people world other man which when
18 ukraine ukrainian india japan fukushima us
19 turkey turkish rebel adl erdogan glacier
20 text god which moon body silver
21 fake earthquake california rodgers jay quake
22 eddie government people country would which
23 which would egypt britain news report
24 war us syria russia military country
25 israel israeli nuclear zika palestinian resolution
26 life time other year find people
27 herb universe which korea planet x
28 trump election day just go get
29 trump would president donald election clinton
30 immigration immigrant illegal percent migrant new
31 law court judge case federal government
32 food energy oil also use which
33 clinton fbi email investigation hillary comey
34 jews jewish which mars only american
In [7]:
pd.DataFrame([[word for rank, (word, prob) in enumerate(words)]
              for topic_id, words in lda_fake.show_topics(formatted=False, num_words=6, num_topics=35)])
Out[7]:
0 1 2 3 4 5
0 dna video abortion veritas project know
1 al syrian force mosul aleppo isis
2 black obama white people soros america
3 china which alien would there ufo
4 get go ’ just know there
5 russia russian putin nato missile nuclear
6 market cancer product bank study report
7 use drug health company report medical
8 post comment facebook news video result
9 pipeline dakota police water rock protester
10 child school police family when people
11 water study health toxic flint tea
12 clinton campaign hillary email podesta wikileaks
13 clinton assange hillary there trump would
14 obama money bank hillary president make
15 party care kelly insurance obamacare year
16 trump vote clinton election hillary voter
17 people world other man which when
18 ukraine ukrainian india japan fukushima us
19 turkey turkish rebel adl erdogan glacier
20 text god which moon body silver
21 fake earthquake california rodgers jay quake
22 eddie government people country would which
23 which would egypt britain news report
24 war us syria russia military country
25 israel israeli nuclear zika palestinian resolution
26 life time other year find people
27 herb universe which korea planet x
28 trump election day just go get
29 trump would president donald election clinton
30 immigration immigrant illegal percent migrant new
31 law court judge case federal government
32 food energy oil also use which
33 clinton fbi email investigation hillary comey
34 jews jewish which mars only american
In [9]:
lda_fake.save('./fake_news_35.lda')

Eye Ballin'

  • usually ML models are evaluated and improved based on a scoring function whose gradient can be followed to a hopefully global minimum
  • unsupervised models are tricky to evaluate because there usually isn't a suitable error function to optimise
  • the unsupervised models still come with hyperparameters so how do you know when you've set them correctly
  • furthermore, how do you know if model A is better than model B

Termite

The visualisation in the Termite paper look very promising, but I've been unable to run the code. The original project has been split into two separate projects a data server and a visualisation client. Unfortunately the data server uses an unknown data format in SQLite databases, and the host server where the data sets ought to be is not operational anymore and the project hasn't been maintained since 2014.

The project also relies on web2py which at the moment only supports python 2 and there doesn't seem to be any interest in porting it to python 3.

Anyhow, it would seem to be possible to run the project under a python 2 environment.

  • modify read_gensim.py to add --sentence-splitter cmd arg
  • modify bin/apps/SplitSentences.py to have an extra param for sentence_splitter jar location
  • update code for gensim API breaking changes
    • bin/readers/GensimReader.py line 47 ldamodel.show_topics
    • bin/readers/GensimReader.py line 51 topic/term distribution does not neet enumerate anymore
    • bin/readers/GensimReader.py line 52 swap term and value around - they are the wrong way around

termite makes a lot of assumptions about paths, one needs to be quite careful what the root directory is for running the commands

In [22]:
import sys

sys.path.append('/home/matti/termite-data-server/bin/')

from modellers import GensimLDA
In [29]:
import re
df_fake['text_oneline'] = df_fake.text.apply(lambda s: re.sub(r'\s+', ' ', str(s)))
In [30]:
df_fake[['uuid', 'text_oneline']].to_csv('./fakenews.termite.tsv', sep='\t', header=False, index=False)
In [82]:
py27 = '/home/matti/miniconda3/envs/py27/bin/python'
termite_server_root = '/home/matti/termite-data-server/'

First we need to import the corpus into termite's own special SQLite format

In [76]:
!mkdir termite;\
    cp ./fakenews.termite.tsv ./termite;
    cd termite; $py27 /home/matti/termite-data-server/bin/import_corpus.py ./db ./fakenews.termite.tsv

mkdir: cannot create directory ‘termite’: File exists
WARNING:root:Unable to import plural rules: No module named plural_rules
Importing file [./fakenews.termite.tsv] into database [./db/corpus.db]

Then we need to export that SQLite DB back into a text corpus, there's some magic file names and path structures that happens here so you can't just use the original file

In [79]:
!cd termite; mkdir corpus; $py27 /home/matti/termite-data-server/bin/export_corpus.py ./db ./corpus/corpus.txt

WARNING:root:Unable to import plural rules: No module named plural_rules
Exporting database [./db/corpus.db] to file [./corpus/corpus.txt]

Then train the LDAModel, is should be possible to skip this and just use any model trained with gensim

In [81]:
%capture
!cd termite; $py27 /home/matti/termite-data-server/bin/train_gensim.py --overwrite ./corpus ./models/

--------------------------------------------------------------------------------
Training an LDA topic model using gensim...
       corpus = ./corpus/corpus.txt
        model = ./models/
  token_regex = \w{3,}
       topics = 20
       passes = 1
--------------------------------------------------------------------------------
using symmetric alpha at 0.05
using symmetric eta at 3.17268948888e-05
using serial LDA version on this node
running online (single-pass) LDA training, 20 topics, 1 passes over the supplied corpus of 12357 documents, updating model once every 2000 documents, evaluating perplexity every 12357 documents, iterating 50x with a convergence threshold of 0.001000
too few updates, training might not converge; consider increasing the number of passes or iterations to improve accuracy
PROGRESS: pass 0, at document #2000/12357
merging changes from 2000 documents into a model of 12357 documents
topic #0 (0.050): 0.003*"black" + 0.003*"obama" + 0.003*"syria" + 0.002*"white" + 0.002*"party" + 0.002*"military" + 0.002*"great" + 0.002*"syrian" + 0.002*"anti" + 0.002*"russia"
topic #14 (0.050): 0.003*"party" + 0.003*"syria" + 0.002*"isis" + 0.002*"house" + 0.002*"russia" + 0.002*"city" + 0.002*"military" + 0.002*"white" + 0.002*"democratic" + 0.002*"power"
topic #12 (0.050): 0.002*"black" + 0.002*"obama" + 0.002*"power" + 0.002*"white" + 0.002*"things" + 0.002*"police" + 0.002*"fbi" + 0.002*"city" + 0.002*"change" + 0.001*"vote"
topic #13 (0.050): 0.004*"syria" + 0.003*"russia" + 0.003*"syrian" + 0.003*"black" + 0.003*"police" + 0.002*"military" + 0.002*"obama" + 0.002*"power" + 0.002*"white" + 0.002*"russian"
topic #17 (0.050): 0.004*"white" + 0.003*"syria" + 0.002*"party" + 0.002*"obama" + 0.002*"israel" + 0.002*"house" + 0.002*"email" + 0.002*"russia" + 0.002*"iran" + 0.002*"foreign"
topic diff=7.126265, rho=1.000000
PROGRESS: pass 0, at document #4000/12357
merging changes from 2000 documents into a model of 12357 documents
topic #3 (0.050): 0.003*"power" + 0.003*"black" + 0.003*"white" + 0.002*"law" + 0.002*"fbi" + 0.002*"americans" + 0.002*"emails" + 0.002*"economic" + 0.002*"article" + 0.002*"life"
topic #8 (0.050): 0.008*"email" + 0.004*"emails" + 0.003*"obama" + 0.002*"law" + 0.002*"follow" + 0.002*"information" + 0.002*"saudi" + 0.002*"russia" + 0.002*"000" + 0.002*"wikileaks"
topic #4 (0.050): 0.003*"assange" + 0.003*"nuclear" + 0.002*"power" + 0.002*"control" + 0.002*"weapons" + 0.002*"russia" + 0.002*"high" + 0.002*"wikileaks" + 0.002*"facebook" + 0.001*"department"
topic #16 (0.050): 0.006*"stockman" + 0.006*"party" + 0.004*"obama" + 0.004*"black" + 0.003*"police" + 0.003*"democratic" + 0.003*"social" + 0.002*"left" + 0.002*"white" + 0.002*"david"
topic #9 (0.050): 0.006*"fbi" + 0.003*"department" + 0.002*"emails" + 0.002*"including" + 0.002*"vote" + 0.002*"party" + 0.002*"weiner" + 0.002*"democratic" + 0.002*"saudi" + 0.002*"officials"
topic diff=2.843640, rho=0.707107
PROGRESS: pass 0, at document #6000/12357
merging changes from 2000 documents into a model of 12357 documents
topic #18 (0.050): 0.005*"obama" + 0.002*"women" + 0.002*"great" + 0.002*"military" + 0.002*"history" + 0.002*"party" + 0.002*"life" + 0.002*"house" + 0.002*"power" + 0.002*"white"
topic #8 (0.050): 0.009*"email" + 0.005*"emails" + 0.003*"obama" + 0.003*"wikileaks" + 0.003*"saudi" + 0.003*"information" + 0.003*"podesta" + 0.002*"law" + 0.002*"000" + 0.002*"2015"
topic #19 (0.050): 0.007*"vote" + 0.006*"obama" + 0.004*"voting" + 0.004*"voters" + 0.004*"white" + 0.004*"republican" + 0.003*"party" + 0.003*"presidential" + 0.003*"candidate" + 0.003*"podesta"
topic #5 (0.050): 0.008*"russia" + 0.005*"russian" + 0.003*"putin" + 0.003*"syria" + 0.002*"military" + 0.002*"nuclear" + 0.002*"foreign" + 0.002*"anti" + 0.002*"israel" + 0.002*"obama"
topic #4 (0.050): 0.006*"health" + 0.005*"brain" + 0.004*"assange" + 0.004*"nuclear" + 0.003*"widget" + 0.003*"food" + 0.003*"medical" + 0.002*"life" + 0.002*"fat" + 0.002*"cancer"
topic diff=1.850195, rho=0.577350
PROGRESS: pass 0, at document #8000/12357
merging changes from 2000 documents into a model of 12357 documents
topic #2 (0.050): 0.003*"ukraine" + 0.003*"rights" + 0.003*"party" + 0.003*"anti" + 0.002*"ukrainian" + 0.002*"left" + 0.002*"saakashvili" + 0.002*"order" + 0.002*"british" + 0.002*"jewish"
topic #17 (0.050): 0.012*"israel" + 0.006*"iran" + 0.004*"israeli" + 0.004*"jerusalem" + 0.003*"palestinian" + 0.003*"jewish" + 0.003*"minister" + 0.003*"obama" + 0.003*"foreign" + 0.003*"resolution"
topic #4 (0.050): 0.008*"health" + 0.008*"food" + 0.005*"cancer" + 0.004*"water" + 0.004*"foods" + 0.004*"brain" + 0.003*"body" + 0.003*"diet" + 0.003*"study" + 0.003*"high"
topic #14 (0.050): 0.005*"comey" + 0.004*"fbi" + 0.004*"moore" + 0.003*"investigation" + 0.003*"house" + 0.002*"party" + 0.002*"airbnb" + 0.002*"she" + 0.002*"power" + 0.002*"khan"
topic #16 (0.050): 0.008*"party" + 0.005*"court" + 0.004*"black" + 0.004*"obama" + 0.004*"supreme" + 0.004*"democratic" + 0.003*"police" + 0.003*"rights" + 0.002*"republican" + 0.002*"white"
topic diff=1.401028, rho=0.500000
PROGRESS: pass 0, at document #10000/12357
merging changes from 2000 documents into a model of 12357 documents
topic #14 (0.050): 0.005*"onion" + 0.004*"sheeple" + 0.004*"automatically" + 0.004*"text" + 0.003*"republish" + 0.003*"comey" + 0.003*"moore" + 0.003*"spam" + 0.003*"star" + 0.003*"fbi"
topic #15 (0.050): 0.013*"police" + 0.009*"water" + 0.006*"pipeline" + 0.005*"dakota" + 0.005*"law" + 0.004*"federal" + 0.004*"standing" + 0.004*"rock" + 0.003*"land" + 0.003*"protesters"
topic #5 (0.050): 0.017*"russia" + 0.010*"russian" + 0.008*"putin" + 0.004*"foreign" + 0.003*"ukraine" + 0.003*"obama" + 0.003*"policy" + 0.003*"nuclear" + 0.003*"military" + 0.003*"europe"
topic #18 (0.050): 0.008*"obama" + 0.003*"space" + 0.002*"god" + 0.002*"women" + 0.002*"great" + 0.002*"house" + 0.002*"man" + 0.002*"history" + 0.002*"earth" + 0.002*"kelly"
topic #16 (0.050): 0.012*"obama" + 0.007*"court" + 0.007*"party" + 0.006*"supreme" + 0.004*"black" + 0.004*"police" + 0.004*"constitution" + 0.004*"white" + 0.004*"rights" + 0.003*"house"
topic diff=1.336365, rho=0.447214
PROGRESS: pass 0, at document #12000/12357
merging changes from 2000 documents into a model of 12357 documents
topic #9 (0.050): 0.007*"department" + 0.006*"soros" + 0.006*"foundation" + 0.005*"law" + 0.004*"officials" + 0.004*"fbi" + 0.003*"federal" + 0.003*"secret" + 0.003*"million" + 0.003*"attorney"
topic #19 (0.050): 0.009*"obama" + 0.008*"vote" + 0.006*"voters" + 0.005*"presidential" + 0.005*"voting" + 0.005*"white" + 0.005*"republican" + 0.004*"party" + 0.004*"candidate" + 0.004*"house"
topic #12 (0.050): 0.004*"love" + 0.003*"life" + 0.003*"energy" + 0.003*"things" + 0.003*"human" + 0.003*"self" + 0.002*"earth" + 0.002*"control" + 0.002*"power" + 0.002*"feel"
topic #3 (0.050): 0.004*"power" + 0.003*"obamacare" + 0.003*"americans" + 0.002*"care" + 0.002*"global" + 0.002*"change" + 0.002*"pay" + 0.002*"order" + 0.002*"economic" + 0.002*"white"
topic #16 (0.050): 0.009*"obama" + 0.008*"court" + 0.007*"party" + 0.006*"supreme" + 0.005*"immigration" + 0.004*"black" + 0.004*"rights" + 0.004*"constitution" + 0.004*"police" + 0.004*"white"
topic diff=1.076053, rho=0.408248
-9.063 per-word bound, 535.0 perplexity estimate based on a held-out corpus of 357 documents with 68794 words
PROGRESS: pass 0, at document #12357/12357
merging changes from 357 documents into a model of 12357 documents
topic #14 (0.050): 0.009*"moore" + 0.004*"violence" + 0.003*"manchanda" + 0.003*"onion" + 0.003*"star" + 0.003*"joe" + 0.003*"sheeple" + 0.003*"foster" + 0.003*"musket" + 0.003*"8th"
topic #7 (0.050): 0.010*"com" + 0.006*"www" + 0.006*"facebook" + 0.005*"http" + 0.005*"information" + 0.004*"data" + 0.004*"google" + 0.003*"star" + 0.003*"content" + 0.003*"internet"
topic #16 (0.050): 0.015*"court" + 0.010*"supreme" + 0.009*"eddie" + 0.008*"judges" + 0.007*"party" + 0.007*"obama" + 0.006*"marriage" + 0.006*"law" + 0.005*"immigration" + 0.004*"unelected"
topic #19 (0.050): 0.009*"obama" + 0.009*"vote" + 0.006*"voting" + 0.006*"voters" + 0.005*"presidential" + 0.005*"republican" + 0.005*"democratic" + 0.004*"white" + 0.004*"candidate" + 0.004*"party"
topic #1 (0.050): 0.005*"life" + 0.004*"women" + 0.004*"children" + 0.003*"man" + 0.003*"old" + 0.003*"family" + 0.003*"child" + 0.003*"father" + 0.002*"home" + 0.002*"things"
topic diff=0.888690, rho=0.377964
Saving dictionary to disk: ./models//dictionary.gensim
Saving corpus to disk: ./models//corpus.gensim
/home/matti/miniconda3/envs/py27/lib/python2.7/site-packages/gensim/interfaces.py:60: UserWarning: corpus.save() stores only the (tiny) iteration object; to serialize the actual corpus content, use e.g. MmCorpus.serialize(corpus)
  warnings.warn("corpus.save() stores only the (tiny) iteration object; "
Saving model to disk: ./models//lda.gensim

Finally, read in the trained gensim LDA model to termite creating all the necessary data structures for the visualisations to work. This computes, among other thigs, term collocations ($N^2$) so it's going to take a while to run, especially for large vocabularies.

If you set all the paths consistently during the previous steps, this should just work. If not, it's likely there will be some FileNotFound errors.

In [88]:
%capture
!cd termite; cp -r $termite_server_root/tools ./; $py27 /home/matti/termite-data-server/bin/read_gensim.py --overwrite\
  --sentence-split /home/matti/termite-data-server/utils/corenlp/SentenceSplitter.jar\
  gensim_termite ./models/ ./corpus ./db

WARNING:root:Unable to import plural rules: No module named plural_rules
--------------------------------------------------------------------------------
INFO:termite:--------------------------------------------------------------------------------
Import a gensim LDA topic model as a web2py application...
INFO:termite:Import a gensim LDA topic model as a web2py application...
           app_name = gensim_termite
INFO:termite:           app_name = gensim_termite
           app_path = apps/gensim_termite
INFO:termite:           app_path = apps/gensim_termite
         model_path = ./models/
INFO:termite:         model_path = ./models/
    corpus_filename = ./corpus/corpus.txt
INFO:termite:    corpus_filename = ./corpus/corpus.txt
  database_filename = ./db/corpus.db
INFO:termite:  database_filename = ./db/corpus.db
--------------------------------------------------------------------------------
INFO:termite:--------------------------------------------------------------------------------
Creating app: gensim_termite [apps/temp_20170623_234907_893474_8228]
INFO:termite:Creating app: gensim_termite [apps/temp_20170623_234907_893474_8228]
Creating folder: [apps/temp_20170623_234907_893474_8228/data]
INFO:termite:Creating folder: [apps/temp_20170623_234907_893474_8228/data]
Creating folder: [apps/temp_20170623_234907_893474_8228/databases]
INFO:termite:Creating folder: [apps/temp_20170623_234907_893474_8228/databases]
Linking folder: [apps/temp_20170623_234907_893474_8228/models]
INFO:termite:Linking folder: [apps/temp_20170623_234907_893474_8228/models]
Linking folder: [apps/temp_20170623_234907_893474_8228/views]
INFO:termite:Linking folder: [apps/temp_20170623_234907_893474_8228/views]
Linking folder: [apps/temp_20170623_234907_893474_8228/controllers]
INFO:termite:Linking folder: [apps/temp_20170623_234907_893474_8228/controllers]
Linking folder: [apps/temp_20170623_234907_893474_8228/static]
INFO:termite:Linking folder: [apps/temp_20170623_234907_893474_8228/static]
Linking folder: [apps/temp_20170623_234907_893474_8228/modules]
INFO:termite:Linking folder: [apps/temp_20170623_234907_893474_8228/modules]
Creating file: [apps/temp_20170623_234907_893474_8228/__init__.py]
INFO:termite:Creating file: [apps/temp_20170623_234907_893474_8228/__init__.py]
Copying [./db/corpus.db] --> [apps/temp_20170623_234907_893474_8228/databases/corpus.db]
INFO:termite:Copying [./db/corpus.db] --> [apps/temp_20170623_234907_893474_8228/databases/corpus.db]
Copying [./corpus/corpus.txt] --> [apps/temp_20170623_234907_893474_8228/data/corpus.txt]
INFO:termite:Copying [./corpus/corpus.txt] --> [apps/temp_20170623_234907_893474_8228/data/corpus.txt]
Extracting [./corpus/corpus.txt] --> [apps/temp_20170623_234907_893474_8228/data/sentences.txt]
INFO:termite:Extracting [./corpus/corpus.txt] --> [apps/temp_20170623_234907_893474_8228/data/sentences.txt]
Preparing the Stanford CoreNLP pipeline...
DEBUG:termite:Preparing the Stanford CoreNLP pipeline...
Adding annotator tokenize
DEBUG:termite:Adding annotator tokenize
Adding annotator ssplit
DEBUG:termite:Adding annotator ssplit
Processing corpus: [./corpus/corpus.txt] -> [apps/temp_20170623_234907_893474_8228/data/sentences.txt]
DEBUG:termite:Processing corpus: [./corpus/corpus.txt] -> [apps/temp_20170623_234907_893474_8228/data/sentences.txt]
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ₨ (U+20A8, decimal: 8360)
DEBUG:termite:WARNING: Untokenizable: ₨ (U+20A8, decimal: 8360)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ‒ (U+2012, decimal: 8210)
DEBUG:termite:WARNING: Untokenizable: ‒ (U+2012, decimal: 8210)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83E, decimal: 55358)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83E, decimal: 55358)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: œ (U+9C, decimal: 156)
DEBUG:termite:WARNING: Untokenizable: œ (U+9C, decimal: 156)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: Ÿ (U+9F, decimal: 159)
DEBUG:termite:WARNING: Untokenizable: Ÿ (U+9F, decimal: 159)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+FFFC, decimal: 65532)
DEBUG:termite:WARNING: Untokenizable:  (U+FFFC, decimal: 65532)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: Ÿ (U+9F, decimal: 159)
DEBUG:termite:WARNING: Untokenizable: Ÿ (U+9F, decimal: 159)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: Ÿ (U+9F, decimal: 159)
DEBUG:termite:WARNING: Untokenizable: Ÿ (U+9F, decimal: 159)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: Ÿ (U+9F, decimal: 159)
DEBUG:termite:WARNING: Untokenizable: Ÿ (U+9F, decimal: 159)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: œ (U+9C, decimal: 156)
DEBUG:termite:WARNING: Untokenizable: œ (U+9C, decimal: 156)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: œ (U+9C, decimal: 156)
DEBUG:termite:WARNING: Untokenizable: œ (U+9C, decimal: 156)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: œ (U+9C, decimal: 156)
DEBUG:termite:WARNING: Untokenizable: œ (U+9C, decimal: 156)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: œ (U+9C, decimal: 156)
DEBUG:termite:WARNING: Untokenizable: œ (U+9C, decimal: 156)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: œ (U+9C, decimal: 156)
DEBUG:termite:WARNING: Untokenizable: œ (U+9C, decimal: 156)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: œ (U+9C, decimal: 156)
DEBUG:termite:WARNING: Untokenizable: œ (U+9C, decimal: 156)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: œ (U+9C, decimal: 156)
DEBUG:termite:WARNING: Untokenizable: œ (U+9C, decimal: 156)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: œ (U+9C, decimal: 156)
DEBUG:termite:WARNING: Untokenizable: œ (U+9C, decimal: 156)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: œ (U+9C, decimal: 156)
DEBUG:termite:WARNING: Untokenizable: œ (U+9C, decimal: 156)
Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:08 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: œ (U+9C, decimal: 156)
DEBUG:termite:WARNING: Untokenizable: œ (U+9C, decimal: 156)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ⁉ (U+2049, decimal: 8265)
DEBUG:termite:WARNING: Untokenizable: ⁉ (U+2049, decimal: 8265)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+9D, decimal: 157)
DEBUG:termite:WARNING: Untokenizable:  (U+9D, decimal: 157)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+9D, decimal: 157)
DEBUG:termite:WARNING: Untokenizable:  (U+9D, decimal: 157)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+9D, decimal: 157)
DEBUG:termite:WARNING: Untokenizable:  (U+9D, decimal: 157)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+9D, decimal: 157)
DEBUG:termite:WARNING: Untokenizable:  (U+9D, decimal: 157)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+9D, decimal: 157)
DEBUG:termite:WARNING: Untokenizable:  (U+9D, decimal: 157)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+9D, decimal: 157)
DEBUG:termite:WARNING: Untokenizable:  (U+9D, decimal: 157)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+9D, decimal: 157)
DEBUG:termite:WARNING: Untokenizable:  (U+9D, decimal: 157)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+9D, decimal: 157)
DEBUG:termite:WARNING: Untokenizable:  (U+9D, decimal: 157)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+9D, decimal: 157)
DEBUG:termite:WARNING: Untokenizable:  (U+9D, decimal: 157)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+9D, decimal: 157)
DEBUG:termite:WARNING: Untokenizable:  (U+9D, decimal: 157)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+9D, decimal: 157)
DEBUG:termite:WARNING: Untokenizable:  (U+9D, decimal: 157)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+9D, decimal: 157)
DEBUG:termite:WARNING: Untokenizable:  (U+9D, decimal: 157)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+81, decimal: 129)
DEBUG:termite:WARNING: Untokenizable:  (U+81, decimal: 129)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83E, decimal: 55358)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83E, decimal: 55358)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: œ (U+9C, decimal: 156)
DEBUG:termite:WARNING: Untokenizable: œ (U+9C, decimal: 156)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: œ (U+9C, decimal: 156)
DEBUG:termite:WARNING: Untokenizable: œ (U+9C, decimal: 156)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: œ (U+9C, decimal: 156)
DEBUG:termite:WARNING: Untokenizable: œ (U+9C, decimal: 156)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: Ÿ (U+9F, decimal: 159)
DEBUG:termite:WARNING: Untokenizable: Ÿ (U+9F, decimal: 159)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: Ÿ (U+9F, decimal: 159)
DEBUG:termite:WARNING: Untokenizable: Ÿ (U+9F, decimal: 159)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: Ÿ (U+9F, decimal: 159)
DEBUG:termite:WARNING: Untokenizable: Ÿ (U+9F, decimal: 159)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ️ (U+FE0F, decimal: 65039)
DEBUG:termite:WARNING: Untokenizable: ️ (U+FE0F, decimal: 65039)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: • (U+95, decimal: 149)
DEBUG:termite:WARNING: Untokenizable: • (U+95, decimal: 149)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:09 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: œ (U+9C, decimal: 156)
DEBUG:termite:WARNING: Untokenizable: œ (U+9C, decimal: 156)
Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: Ÿ (U+9F, decimal: 159)
DEBUG:termite:WARNING: Untokenizable: Ÿ (U+9F, decimal: 159)
Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ƒ (U+83, decimal: 131)
DEBUG:termite:WARNING: Untokenizable: ƒ (U+83, decimal: 131)
Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ™ (U+99, decimal: 153)
DEBUG:termite:WARNING: Untokenizable: ™ (U+99, decimal: 153)
Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ‌ (U+200C, decimal: 8204)
DEBUG:termite:WARNING: Untokenizable: ‌ (U+200C, decimal: 8204)
Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+F4A9, decimal: 62633)
DEBUG:termite:WARNING: Untokenizable:  (U+F4A9, decimal: 62633)
Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ƒ (U+83, decimal: 131)
DEBUG:termite:WARNING: Untokenizable: ƒ (U+83, decimal: 131)
Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: Ÿ (U+9F, decimal: 159)
DEBUG:termite:WARNING: Untokenizable: Ÿ (U+9F, decimal: 159)
Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ƒ (U+83, decimal: 131)
DEBUG:termite:WARNING: Untokenizable: ƒ (U+83, decimal: 131)
Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+8D, decimal: 141)
DEBUG:termite:WARNING: Untokenizable:  (U+8D, decimal: 141)
Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: š (U+9A, decimal: 154)
DEBUG:termite:WARNING: Untokenizable: š (U+9A, decimal: 154)
Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ‒ (U+2012, decimal: 8210)
DEBUG:termite:WARNING: Untokenizable: ‒ (U+2012, decimal: 8210)
Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ‒ (U+2012, decimal: 8210)
DEBUG:termite:WARNING: Untokenizable: ‒ (U+2012, decimal: 8210)
Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+81, decimal: 129)
DEBUG:termite:WARNING: Untokenizable:  (U+81, decimal: 129)
Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:10 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+F0B7, decimal: 61623)
DEBUG:termite:WARNING: Untokenizable:  (U+F0B7, decimal: 61623)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ‒ (U+2012, decimal: 8210)
DEBUG:termite:WARNING: Untokenizable: ‒ (U+2012, decimal: 8210)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+9D, decimal: 157)
DEBUG:termite:WARNING: Untokenizable:  (U+9D, decimal: 157)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+9D, decimal: 157)
DEBUG:termite:WARNING: Untokenizable:  (U+9D, decimal: 157)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+9D, decimal: 157)
DEBUG:termite:WARNING: Untokenizable:  (U+9D, decimal: 157)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+9D, decimal: 157)
DEBUG:termite:WARNING: Untokenizable:  (U+9D, decimal: 157)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+9D, decimal: 157)
DEBUG:termite:WARNING: Untokenizable:  (U+9D, decimal: 157)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+9D, decimal: 157)
DEBUG:termite:WARNING: Untokenizable:  (U+9D, decimal: 157)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+9D, decimal: 157)
DEBUG:termite:WARNING: Untokenizable:  (U+9D, decimal: 157)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+9D, decimal: 157)
DEBUG:termite:WARNING: Untokenizable:  (U+9D, decimal: 157)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+9D, decimal: 157)
DEBUG:termite:WARNING: Untokenizable:  (U+9D, decimal: 157)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+F50D, decimal: 62733)
DEBUG:termite:WARNING: Untokenizable:  (U+F50D, decimal: 62733)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ₹ (U+20B9, decimal: 8377)
DEBUG:termite:WARNING: Untokenizable: ₹ (U+20B9, decimal: 8377)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ₹ (U+20B9, decimal: 8377)
DEBUG:termite:WARNING: Untokenizable: ₹ (U+20B9, decimal: 8377)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ₹ (U+20B9, decimal: 8377)
DEBUG:termite:WARNING: Untokenizable: ₹ (U+20B9, decimal: 8377)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+F1FA, decimal: 61946)
DEBUG:termite:WARNING: Untokenizable:  (U+F1FA, decimal: 61946)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+F6A8, decimal: 63144)
DEBUG:termite:WARNING: Untokenizable:  (U+F6A8, decimal: 63144)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+F1FA, decimal: 61946)
DEBUG:termite:WARNING: Untokenizable:  (U+F1FA, decimal: 61946)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+F682, decimal: 63106)
DEBUG:termite:WARNING: Untokenizable:  (U+F682, decimal: 63106)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+F1FA, decimal: 61946)
DEBUG:termite:WARNING: Untokenizable:  (U+F1FA, decimal: 61946)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable:  (U+F602, decimal: 62978)
DEBUG:termite:WARNING: Untokenizable:  (U+F602, decimal: 62978)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83E, decimal: 55358)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83E, decimal: 55358)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ️ (U+FE0F, decimal: 65039)
DEBUG:termite:WARNING: Untokenizable: ️ (U+FE0F, decimal: 65039)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83E, decimal: 55358)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83E, decimal: 55358)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ️ (U+FE0F, decimal: 65039)
DEBUG:termite:WARNING: Untokenizable: ️ (U+FE0F, decimal: 65039)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:11 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:12 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:12 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
Jun 23, 2017 11:49:12 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:12 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
Jun 23, 2017 11:49:12 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:12 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: � (U+FFFD, decimal: 65533)
DEBUG:termite:WARNING: Untokenizable: � (U+FFFD, decimal: 65533)
Jun 23, 2017 11:49:12 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:12 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83D, decimal: 55357)
Jun 23, 2017 11:49:12 PM edu.stanford.nlp.process.PTBLexer next
DEBUG:termite:Jun 23, 2017 11:49:12 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
DEBUG:termite:WARNING: Untokenizable: ? (U+D83C, decimal: 55356)
Copying [./models/] --> [apps/temp_20170623_234907_893474_8228/data/gensim-lda]
INFO:termite:Copying [./models/] --> [apps/temp_20170623_234907_893474_8228/data/gensim-lda]
Computing bag-of-words statistics
INFO:termite:Computing bag-of-words statistics
          token_regex = \w{3,}
INFO:termite:          token_regex = \w{3,}
             min_freq = 5
INFO:termite:             min_freq = 5
         min_doc_freq = 3
INFO:termite:         min_doc_freq = 3
       max_freq_count = 4000
INFO:termite:       max_freq_count = 4000
    max_co_freq_count = 100000
INFO:termite:    max_co_freq_count = 100000
Computing document-level statistics...
INFO:termite:Computing document-level statistics...
    Loading corpus: apps/temp_20170623_234907_893474_8228/data/corpus.txt
DEBUG:termite:    Loading corpus: apps/temp_20170623_234907_893474_8228/data/corpus.txt
    Computing term freqs (12357 docs)...
INFO:termite:    Computing term freqs (12357 docs)...
    Computing term co-occurrences (12357 docs)...
INFO:termite:    Computing term co-occurrences (12357 docs)...
    Saving term_texts (4000 terms)...
DEBUG:termite:    Saving term_texts (4000 terms)...
        inserting 4000 rows...
DEBUG:termite:        inserting 4000 rows...
    Saving term_freqs (4000 terms)...
DEBUG:termite:    Saving term_freqs (4000 terms)...
        inserting 4000 rows...
DEBUG:termite:        inserting 4000 rows...
    Saving term_probs (4000 terms)...
DEBUG:termite:    Saving term_probs (4000 terms)...
        inserting 4000 rows...
DEBUG:termite:        inserting 4000 rows...
    Saving term_doc_freqs (4000 terms)...
DEBUG:termite:    Saving term_doc_freqs (4000 terms)...
        inserting 4000 rows...
DEBUG:termite:        inserting 4000 rows...
    Saving term_co_freqs (100000 term pairs)...
DEBUG:termite:    Saving term_co_freqs (100000 term pairs)...
        inserting 100000 rows...
DEBUG:termite:        inserting 100000 rows...
    Saving term_co_probs (100000 term pairs)...
DEBUG:termite:    Saving term_co_probs (100000 term pairs)...
        inserting 100000 rows...
DEBUG:termite:        inserting 100000 rows...
    Saving term_g2 (100000 term pairs)...
DEBUG:termite:    Saving term_g2 (100000 term pairs)...
        inserting 100000 rows...
DEBUG:termite:        inserting 100000 rows...
Computing sentence-level term statistics...
INFO:termite:Computing sentence-level term statistics...
    Loading corpus: apps/temp_20170623_234907_893474_8228/data/sentences.txt
DEBUG:termite:    Loading corpus: apps/temp_20170623_234907_893474_8228/data/sentences.txt
    Computing term freqs (369703 docs)...
INFO:termite:    Computing term freqs (369703 docs)...
    Computing term co-occurrences (369703 docs)...
INFO:termite:    Computing term co-occurrences (369703 docs)...
    Saving sentences_co_freqs (100000 term pairs)...
DEBUG:termite:    Saving sentences_co_freqs (100000 term pairs)...
        inserting 100000 rows...
DEBUG:termite:        inserting 100000 rows...
    Saving sentences_co_probs (100000 term pairs)...
DEBUG:termite:    Saving sentences_co_probs (100000 term pairs)...
        inserting 100000 rows...
DEBUG:termite:        inserting 100000 rows...
    Saving sentences_g2 (100000 term pairs)...
DEBUG:termite:    Saving sentences_g2 (100000 term pairs)...
        inserting 100000 rows...
DEBUG:termite:        inserting 100000 rows...
('-->', 'apps/temp_20170623_234907_893474_8228/data/gensim-lda/corpus.gensim')
Reading gensim LDA output...
INFO:termite:Reading gensim LDA output...
    Loading dictionary: apps/temp_20170623_234907_893474_8228/data/gensim-lda/dictionary.gensim
DEBUG:termite:    Loading dictionary: apps/temp_20170623_234907_893474_8228/data/gensim-lda/dictionary.gensim
    Loading corpus: apps/temp_20170623_234907_893474_8228/data/gensim-lda/corpus.gensim
DEBUG:termite:    Loading corpus: apps/temp_20170623_234907_893474_8228/data/gensim-lda/corpus.gensim
('||-->', 'apps/temp_20170623_234907_893474_8228/data/gensim-lda/corpus.gensim')
<class 'modellers.GensimLDA.GensimTermiteCorpusReader'>
    Loading model: apps/temp_20170623_234907_893474_8228/data/gensim-lda/lda.gensim
DEBUG:termite:    Loading model: apps/temp_20170623_234907_893474_8228/data/gensim-lda/lda.gensim
Writing to database...
INFO:termite:Writing to database...
    Saving term_topic_matrix...
DEBUG:termite:    Saving term_topic_matrix...
    Saving doc_topic_matrix...
DEBUG:termite:    Saving doc_topic_matrix...
    Retrieving terms, documents, and topics...
DEBUG:termite:    Retrieving terms, documents, and topics...
    Retrieving top terms and top documents...
DEBUG:termite:    Retrieving top terms and top documents...
    Saving terms...
DEBUG:termite:    Saving terms...
    Saving docs...
DEBUG:termite:    Saving docs...
    Saving topics...
DEBUG:termite:    Saving topics...
Computing derived LDA topic model statistics...
INFO:termite:Computing derived LDA topic model statistics...
    max_co_topic_count = 10000
INFO:termite:    max_co_topic_count = 10000
    Loading doc_topic_matrix...
DEBUG:termite:    Loading doc_topic_matrix...
    Computing topic cooccurrences...
DEBUG:termite:    Computing topic cooccurrences...
    Computing topic covariance...
DEBUG:termite:    Computing topic covariance...
    Saving topic_covariance...
DEBUG:termite:    Saving topic_covariance...
        inserting 400 rows...
DEBUG:termite:        inserting 400 rows...
Moving app into place: gensim_termite [apps/temp_20170623_234907_893474_8228] -> [apps/gensim_termite]
INFO:termite:Moving app into place: gensim_termite [apps/temp_20170623_234907_893474_8228] -> [apps/gensim_termite]

To start the server and see the visualisations

In [91]:
!$py27 $termite_server_root/web2py/web2py.py

web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2017
Version 2.9.5-stable+timestamp.2014.03.16.02.35.39
Database drivers available: SQLite(sqlite3), MySQL(pymysql), PostgreSQL(pg8000), MSSQL(pyodbc), DB2(pyodbc), Teradata(pyodbc), Ingres(pyodbc), IMAP(imaplib)

pyLDAVis

Some of the work from Termite has been integrated into pyLDAVis which is being maintained and has good interoperability with gensim. Below is an interactive visualisation of the fake news model trained earlier. Just to see how informative the visualisation is overall, I'll train another model on the same dataset but increaase the number of topics quite a lot.

For a good description of what you see in the visualisation you can look at the presenation from the creator himself

In [6]:
lda_fake = LdaModel.load('./fake_news_35.lda')
In [15]:
from gensim.models import LdaModel

import pyLDAvis as ldavis
import pyLDAvis.gensim

ldavis.enable_notebook()
prepared_data = ldavis.gensim.prepare(lda_fake, corpus_fake, vocabulary)

with open('./fake_news_35.lda-LDAVIS.json', 'w') as fh:
    fh.write(prepared_data.to_json())

prepared_data

/home/matti/miniconda3/envs/pydatabln17/lib/python3.6/site-packages/pyLDAvis/_prepare.py:387: DeprecationWarning: 
.ix is deprecated. Please use
.loc for label based indexing or
.iloc for positional indexing

See the documentation here:
http://pandas.pydata.org/pandas-docs/stable/indexing.html#deprecate_ix
  topic_term_dists = topic_term_dists.ix[topic_order]
Out[15]: