from theano.sandbox import cuda
cuda.use('gpu2')
Using gpu device 2: GeForce GTX TITAN X (CNMeM is enabled with initial size: 90.0% of memory, cuDNN 4007)
%matplotlib inline
import utils; reload(utils)
from utils import *
from __future__ import division, print_function
Using Theano backend.
from keras.layers import TimeDistributed, Activation
from numpy.random import choice
We haven't really looked into the detail of how this works yet - so this is provided for self-study for those who are interested. We'll look at it closely next week.
path = get_file('nietzsche.txt', origin="https://s3.amazonaws.com/text-datasets/nietzsche.txt")
text = open(path).read().lower()
print('corpus length:', len(text))
corpus length: 600901
!tail {path} -n25
are thinkers who believe in the saints. 144 It stands to reason that this sketch of the saint, made upon the model of the whole species, can be confronted with many opposing sketches that would create a more agreeable impression. There are certain exceptions among the species who distinguish themselves either by especial gentleness or especial humanity, and perhaps by the strength of their own personality. Others are in the highest degree fascinating because certain of their delusions shed a particular glow over their whole being, as is the case with the founder of christianity who took himself for the only begotten son of God and hence felt himself sinless; so that through his imagination--that should not be too harshly judged since the whole of antiquity swarmed with sons of god--he attained the same goal, the sense of complete sinlessness, complete irresponsibility, that can now be attained by every individual through science.--In the same manner I have viewed the saints of India who occupy an intermediate station between the christian saints and the Greek philosophers and hence are not to be regarded as a pure type. Knowledge and science--as far as they existed--and superiority to the rest of mankind by logical discipline and training of the intellectual powers were insisted upon by the Buddhists as essential to sanctity, just as they were denounced by the christian world as the indications of sinfulness.
#path = 'data/wiki/'
#text = open(path+'small.txt').read().lower()
#print('corpus length:', len(text))
#text = text[0:1000000]
corpus length: 137587200
chars = sorted(list(set(text)))
vocab_size = len(chars)+1
print('total chars:', vocab_size)
total chars: 60
chars.insert(0, "\0")
''.join(chars[1:-6])
'\n !"\'(),-.0123456789:;=?[]_abcdefghijklmnopqrstuvwxyz'
char_indices = dict((c, i) for i, c in enumerate(chars))
indices_char = dict((i, c) for i, c in enumerate(chars))
idx = [char_indices[c] for c in text]
idx[:10]
[43, 45, 32, 33, 28, 30, 32, 1, 1, 1]
''.join(indices_char[i] for i in idx[:70])
'preface\n\n\nsupposing that truth is a woman--what then? is there not gro'
maxlen = 40
sentences = []
next_chars = []
for i in range(0, len(idx) - maxlen+1):
sentences.append(idx[i: i + maxlen])
next_chars.append(idx[i+1: i+maxlen+1])
print('nb sequences:', len(sentences))
0 nb sequences: 600862
sentences = np.concatenate([[np.array(o)] for o in sentences[:-2]])
next_chars = np.concatenate([[np.array(o)] for o in next_chars[:-2]])
sentences.shape, next_chars.shape
((600860, 40), (600860, 40))
n_fac = 24
model=Sequential([
Embedding(vocab_size, n_fac, input_length=maxlen),
LSTM(512, input_dim=n_fac,return_sequences=True, dropout_U=0.2, dropout_W=0.2,
consume_less='gpu'),
Dropout(0.2),
LSTM(512, return_sequences=True, dropout_U=0.2, dropout_W=0.2,
consume_less='gpu'),
Dropout(0.2),
TimeDistributed(Dense(vocab_size)),
Activation('softmax')
])
model.compile(loss='sparse_categorical_crossentropy', optimizer=Adam())
def print_example():
seed_string="ethics is a basic foundation of all that"
for i in range(320):
x=np.array([char_indices[c] for c in seed_string[-40:]])[np.newaxis,:]
preds = model.predict(x, verbose=0)[0][-1]
preds = preds/np.sum(preds)
next_char = choice(chars, p=preds)
seed_string = seed_string + next_char
print(seed_string)
model.fit(sentences, np.expand_dims(next_chars,-1), batch_size=64, nb_epoch=1)
Epoch 1/1 600860/600860 [==============================] - 640s - loss: 1.5152
<keras.callbacks.History at 0x7f8cf2398810>
print_example()
ethics is a basic foundation of all thatscrriets sdi ,s lrrbmh fceelsora tec n yiefma cnostencnrut - o pen.htt" oaiosovo stialpts es rb b ea ie ohatnmauyielueysiutlmo,es etfrne oh ohnio iis e.eosme o rdorfdbteirnse ohdnotafi enicron e eietnyn sytt e ptsrdrede httmi ah oo, tdye es r,igyct toehitu abrh ei isiem-r natra lnspamlltefae a cni vuui twgt fatieh
model.fit(sentences, np.expand_dims(next_chars,-1), batch_size=64, nb_epoch=1)
Epoch 1/1 600860/600860 [==============================] - 640s - loss: 1.5152
<keras.callbacks.History at 0x7f8cf2398810>
print_example()
ethics is a basic foundation of all that he maluces indofely and is; pticrast', and re onerog je ivesantamale as whered and ror and kytinf? on chaninn nurdeln--ans prory. heke the pepadinar; anf bom, puntely"" ones to bucf, alcherstol the qisleves: the the wite dadong the gur is prang not galcaula rewinl more by than sic appads not pepow o mee, a more bins c
model.optimizer.lr=0.001
model.fit(sentences, np.expand_dims(next_chars,-1), batch_size=64, nb_epoch=1)
Epoch 1/1 600860/600860 [==============================] - 640s - loss: 1.5152
<keras.callbacks.History at 0x7f8cf2398810>
print_example()
ethics is a basic foundation of all that schools pedhaps a new prisons of the ashamed in which a coverbine estimates of the assumption that one avoid; he will curse about pain: people, he-equally present to the lalier, nature. that he has rendered and henceforth distrain and impulses to perceive that each other former and dangerous, and cannot at the pu
model.optimizer.lr=0.0001
model.fit(sentences, np.expand_dims(next_chars,-1), batch_size=64, nb_epoch=1)
Epoch 1/1 600860/600860 [==============================] - 639s - loss: 1.2892
<keras.callbacks.History at 0x7f8cf167b410>
print_example()
ethics is a basic foundation of all that account has its granitify them. 131. the new "dilence," out of the same light, interpretation thereof: under the "thinking" there, to counter-arguments in the monality, so many language: though all nobilitys of higher impulses, man and hence to everything of seldom man. chapter i. woman decides according the injur
model.save_weights('data/char_rnn.h5')
model.optimizer.lr=0.00001
model.fit(sentences, np.expand_dims(next_chars,-1), batch_size=64, nb_epoch=1)
Epoch 1/1 600860/600860 [==============================] - 640s - loss: 1.2544
<keras.callbacks.History at 0x7f8cf2398f50>
print_example()
ethics is a basic foundation of all that is military contemplation of distance itself is in physician! 249.. in every to strick in the man of disguise and in the will to wind at any progress, the religious estimates of vehapance has a powerful and religious nature of manner, who had the problem of decided expression of his equality, which, sometimes power?
model.fit(sentences, np.expand_dims(next_chars,-1), batch_size=64, nb_epoch=1)
Epoch 1/1 600860/600860 [==============================] - 640s - loss: 1.2218
<keras.callbacks.History at 0x7f8cf2a21590>
print_example()
ethics is a basic foundation of all that the belief in the importance. the employs concerning seriousness and materialism, it is circles which alone is already attained, that he sees also the day after thinking of mankind, brightness, resistance--and after the value of "nature" in order to nevertheless have taken a system of liberal fatalists are willing him
print_example()
ethics is a basic foundation of all that were beought by the temptation of truth--for the rest of a sublime medely and take part of life, which lacks himself the credibility about this, in short, and raise such a gods; and on the other hand, the explanation of the case, as the most ingredient, and insight, and approach as to the peculiarly prolonged "distrus
model.save_weights('data/char_rnn.h5')