So far, word2vec cannot increase the size of vocabulary after initial training. To handle unknown words, not in word2vec vocaburary, you must retrain updated documents over again.
In this tutorial, we introduce gensim new feature, online vocaburary update. This additional feature overcomes the unknown word problems. Despite after initial training, we can continuously add new vocaburary to the pre-trained word2vec model using this online feature.
from gensim.corpora.wikicorpus import WikiCorpus
from gensim.models.word2vec import Word2Vec, LineSentence
from pprint import pprint
from copy import deepcopy
from multiprocessing import cpu_count
from smart_open import smart_open
We use the past and the current version of wiki dump files as online training.
%%bash
wget https://dumps.wikimedia.org/archive/2010/2010-11/enwiki/20101011/enwiki-20101011-pages-articles.xml.bz2
wget https://dumps.wikimedia.org/enwiki/20160820/enwiki-20160820-pages-articles.xml.bz2
To avoid alert when convert old verision of wikipedia dump, you should download alternative wikicorpus.py in my repo.
old, new = [WikiCorpus('enwiki-{}-pages-articles.xml.bz2'.format(ymd)) for ymd in ['20101011', '20160820']]
def write_wiki(wiki, name, titles = []):
with smart_open('{}.wiki'.format(name), 'wb') as f:
wiki.metadata = True
for text, (page_id, title) in wiki.get_texts():
if title not in titles:
f.write(b' '.join(text)+b'\n')
titles.append(title)
return titles
old_titles = write_wiki(old, 'old')
all_titles = write_wiki(new, 'new', old_titles)
oldwiki, newwiki = [LineSentence(f+'.wiki') for f in ['old', 'new']]
At first we train word2vec using "enwiki-20101011-pages-articles.xml.bz2". After that, we update model using "enwiki-20160820-pages-articles.xml.bz2".
%%time
model = Word2Vec(oldwiki, min_count = 0, workers=cpu_count())
# model = Word2Vec.load('oldmodel')
oldmodel = deepcopy(model)
oldmodel.save('oldmodel')
CPU times: user 4h 39min 57s, sys: 1min 28s, total: 4h 41min 25s Wall time: 1h 32min 43s
Note: In recent years, they became the famous idol group not only in Japan. They won many music awards and run world tour.
try:
print(oldmodel.most_similar('babymetal'))
except KeyError as e:
print(e)
"word 'babymetal' not in vocabulary"
To use online word2vec feature, set update=True when you use build_vocab using new documents.
%%time
model.build_vocab(newwiki, update=True)
model.train(newwiki, total_examples=model.corpus_count, epochs=model.iter)
model.save('newmodel')
# model = Word2Vec.load('newmodel')
CPU times: user 2h 53min 51s, sys: 1min 1s, total: 2h 54min 52s Wall time: 57min 24s
By the online training, the size of vocaburaries are increased about 3 millions.
for m in ['oldmodel', 'model']:
print('The vocabulary size of the', m, 'is', len(eval(m).wv.vocab))
The vocabulary size of the oldmodel is 6161170 The vocabulary size of the model is 8469444
try:
pprint(model.most_similar('babymetal'))
except KeyError as e:
print(e)
[('espairsray', 0.7539531588554382), ('crossfaith', 0.7476214170455933), ('mucc', 0.7363666296005249), ('girugamesh', 0.7309226989746094), ('flumpool', 0.7182492017745972), ('gackt', 0.715751051902771), ('jpop', 0.7055245637893677), ('kuroyume', 0.7049269676208496), ('ellegarden', 0.7018687725067139), ('tigertailz', 0.701062023639679)]
In the past, the word, "Zootopia", was used just for an annual summer concert put on by New York top-40 radio station Z100, so that the word, "zootopia", is simillar with music festival.
In 2016, Zootopia is a American 3D computer-animated comedy film released by Walt Disney Pictures. As a result, the word, "zootopia", was often used as Animation films.
w = 'zootopia'
for m in ['oldmodel', 'model']:
print('The count of the word,'+w+', is', eval(m).wv.vocab[w].count, 'in', m)
pprint(eval(m).most_similar(w))
print('')
The count of the word,zootopia, is 24 in oldmodel [('itsekseni', 0.655870258808136), ('baverstam', 0.6502687931060791), ('hachnosas', 0.6450551748275757), ('carrantouhill', 0.631106436252594), ('bugasan', 0.6258121728897095), ('lollapolooza', 0.6192305088043213), ('hutuz', 0.6134281754493713), ('soulico', 0.6122198104858398), ('kabungwe', 0.6060466766357422), ('prischoßhalle', 0.6056506633758545)] The count of the word,zootopia, is 257 in model [('incredibles', 0.7643648386001587), ('antz', 0.7575620412826538), ('spaceballs', 0.7434272766113281), ('pagemaster', 0.730089545249939), ('beetlejuice', 0.7257461547851562), ('coneheads', 0.7239412069320679), ('tarzan', 0.7139339447021484), ('catscratch', 0.7124171257019043), ('boxtrolls', 0.7024375796318054), ('aristocats', 0.7005465030670166)]