Deep Learning Models -- A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.

Model Zoo -- Character RNN for Generating Text

A simple character-level RNN to generate new bits of text based on text from a novel.

In [1]:
%load_ext watermark
%watermark -a 'Sebastian Raschka' -v -p torch

import torch
import torch.nn.functional as F
from torchtext import data
from torchtext import datasets
import time
import random
import unidecode
import string
import random
import re


torch.backends.cudnn.deterministic = True
Sebastian Raschka 

CPython 3.7.1
IPython 7.4.0

torch 1.0.1.post2

General Settings

In [2]:
RANDOM_SEED = 123
torch.manual_seed(RANDOM_SEED)

DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

TEXT_PORTION_SIZE = 200

NUM_ITER = 20000
LEARNING_RATE = 0.005
EMBEDDING_DIM = 100
HIDDEN_DIM = 100
NUM_HIDDEN = 1

Dataset

Download A Tale of Two Cities by Charles Dickens from the Gutenberg Project:

In [3]:
!wget http://www.gutenberg.org/files/98/98-0.txt
--2019-04-26 04:03:36--  http://www.gutenberg.org/files/98/98-0.txt
Resolving www.gutenberg.org (www.gutenberg.org)... 152.19.134.47, 2610:28:3090:3000:0:bad:cafe:47
Connecting to www.gutenberg.org (www.gutenberg.org)|152.19.134.47|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 804335 (785K) [text/plain]
Saving to: ‘98-0.txt.11’

98-0.txt.11         100%[===================>] 785.48K  1.68MB/s    in 0.5s    

2019-04-26 04:03:36 (1.68 MB/s) - ‘98-0.txt.11’ saved [804335/804335]

Convert all characters into ASCII characters provided by string.printable:

In [4]:
string.printable
Out[4]:
'0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&\'()*+,-./:;<=>[email protected][\\]^_`{|}~ \t\n\r\x0b\x0c'
In [5]:
with open('./98-0.txt', 'r') as f:
    textfile = f.read()

# convert special characters
textfile = unidecode.unidecode(textfile)

# strip extra whitespaces
textfile = re.sub(' +',' ', textfile)

TEXT_LENGTH = len(textfile)

print(f'Number of characters in text: {TEXT_LENGTH}')
Number of characters in text: 776058

Divide the text into smaller portions:

In [6]:
random.seed(RANDOM_SEED)

def random_portion(textfile):
    start_index = random.randint(0, TEXT_LENGTH - TEXT_PORTION_SIZE)
    end_index = start_index + TEXT_PORTION_SIZE + 1
    return textfile[start_index:end_index]

print(random_portion(textfile))
 left his saw sticking in the firewood he was cutting, set it in
motion again; the women who had left on a door-step the little pot of
hot ashes, at which she had been trying to soften the pain in her 

Define a function to convert characters into tensors of integers (type long):

In [7]:
def char_to_tensor(text):
    lst = [string.printable.index(c) for c in text]
    tensor = torch.tensor(lst).long()
    return tensor

print(char_to_tensor('abcDEF'))
tensor([10, 11, 12, 39, 40, 41])

Putting it together to make a function that draws random batches for training:

In [ ]:
def draw_random_sample(textfile):    
    text_long = char_to_tensor(random_portion(textfile))
    inputs = text_long[:-1]
    targets = text_long[1:]
    return inputs, targets
In [9]:
draw_random_sample(textfile)
Out[9]:
(tensor([94, 17, 18, 28, 94, 32, 18, 23, 13, 24, 32, 94, 10, 28, 94, 18, 15, 94,
         29, 17, 10, 29, 94, 32, 14, 27, 14, 94, 27, 30, 21, 14, 13, 94, 15, 24,
         27, 94, 15, 18, 16, 30, 27, 14, 28, 94, 29, 24, 24, 73, 94, 10, 23, 13,
         94, 14, 31, 14, 27, 34, 29, 17, 18, 23, 16, 96, 30, 23, 13, 14, 27, 94,
         29, 17, 14, 94, 12, 21, 24, 30, 13, 28, 94, 32, 14, 27, 14, 94, 10, 94,
         28, 30, 22, 75, 96, 96, 63, 43, 10, 21, 21, 24, 10, 62, 63, 94, 28, 10,
         18, 13, 94, 48, 27, 75, 94, 54, 29, 27, 34, 31, 14, 27, 75, 94, 63, 43,
         24, 32, 94, 13, 24, 94, 34, 24, 30, 94, 13, 24, 82, 94, 44, 94, 17, 24,
         25, 14, 94, 34, 24, 30, 94, 10, 27, 14, 94, 32, 14, 21, 21, 62, 63, 96,
         96, 44, 29, 94, 32, 10, 28, 94, 54, 29, 27, 34, 31, 14, 27, 68, 28, 94,
         16, 27, 10, 23, 13, 94, 25, 14, 12, 30, 21, 18, 10, 27, 18, 29, 34, 94,
         29, 17]),
 tensor([17, 18, 28, 94, 32, 18, 23, 13, 24, 32, 94, 10, 28, 94, 18, 15, 94, 29,
         17, 10, 29, 94, 32, 14, 27, 14, 94, 27, 30, 21, 14, 13, 94, 15, 24, 27,
         94, 15, 18, 16, 30, 27, 14, 28, 94, 29, 24, 24, 73, 94, 10, 23, 13, 94,
         14, 31, 14, 27, 34, 29, 17, 18, 23, 16, 96, 30, 23, 13, 14, 27, 94, 29,
         17, 14, 94, 12, 21, 24, 30, 13, 28, 94, 32, 14, 27, 14, 94, 10, 94, 28,
         30, 22, 75, 96, 96, 63, 43, 10, 21, 21, 24, 10, 62, 63, 94, 28, 10, 18,
         13, 94, 48, 27, 75, 94, 54, 29, 27, 34, 31, 14, 27, 75, 94, 63, 43, 24,
         32, 94, 13, 24, 94, 34, 24, 30, 94, 13, 24, 82, 94, 44, 94, 17, 24, 25,
         14, 94, 34, 24, 30, 94, 10, 27, 14, 94, 32, 14, 21, 21, 62, 63, 96, 96,
         44, 29, 94, 32, 10, 28, 94, 54, 29, 27, 34, 31, 14, 27, 68, 28, 94, 16,
         27, 10, 23, 13, 94, 25, 14, 12, 30, 21, 18, 10, 27, 18, 29, 34, 94, 29,
         17, 10]))

Model

In [ ]:
class RNN(torch.nn.Module):
    def __init__(self, input_size, embed_size,
                 hidden_size, output_size, num_layers):
        super(RNN, self).__init__()

        self.num_layers = num_layers
        self.hidden_size = hidden_size
        
        self.embed = torch.nn.Embedding(input_size, hidden_size)
        self.gru = torch.nn.GRU(input_size=embed_size,
                                hidden_size=hidden_size,
                                num_layers=num_layers)
        self.fc = torch.nn.Linear(hidden_size, output_size)
        self.init_hidden = torch.nn.Parameter(torch.zeros(
                                              num_layers, 1, hidden_size))
    
    def forward(self, features, hidden):
        embedded = self.embed(features.view(1, -1))
        output, hidden = self.gru(embedded.view(1, 1, -1), hidden)
        output = self.fc(output.view(1, -1))
        return output, hidden
      
    def init_zero_state(self):
        init_hidden = torch.zeros(self.num_layers, 1, self.hidden_size).to(DEVICE)
        return init_hidden
In [ ]:
torch.manual_seed(RANDOM_SEED)
model = RNN(len(string.printable), EMBEDDING_DIM, HIDDEN_DIM, len(string.printable), NUM_HIDDEN)
model = model.to(DEVICE)
optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)

Training

In [ ]:
def evaluate(model, prime_str='A', predict_len=100, temperature=0.8):
    ## based on https://github.com/spro/practical-pytorch/
    ## blob/master/char-rnn-generation/char-rnn-generation.ipynb

    hidden = model.init_zero_state()
    prime_input = char_to_tensor(prime_str)
    predicted = prime_str

    # Use priming string to "build up" hidden state
    for p in range(len(prime_str) - 1):
        _, hidden = model(prime_input[p].to(DEVICE), hidden.to(DEVICE))
    inp = prime_input[-1]
    
    for p in range(predict_len):
        output, hidden = model(inp.to(DEVICE), hidden.to(DEVICE))
        
        # Sample from the network as a multinomial distribution
        output_dist = output.data.view(-1).div(temperature).exp()
        top_i = torch.multinomial(output_dist, 1)[0]
        
        # Add predicted character to string and use as next input
        predicted_char = string.printable[top_i]
        predicted += predicted_char
        inp = char_to_tensor(predicted_char)

    return predicted
In [13]:
start_time = time.time()
for iteration in range(NUM_ITER):

    
    ### FORWARD AND BACK PROP

    hidden = model.init_zero_state()
    optimizer.zero_grad()
    
    loss = 0.
    inputs, targets = draw_random_sample(textfile)
    inputs, targets = inputs.to(DEVICE), targets.to(DEVICE)
    for c in range(TEXT_PORTION_SIZE):
        outputs, hidden = model(inputs[c], hidden)
        loss += F.cross_entropy(outputs, targets[c].view(1))

    loss /= TEXT_PORTION_SIZE
    loss.backward()
    
    ### UPDATE MODEL PARAMETERS
    optimizer.step()

    ### LOGGING
    with torch.set_grad_enabled(False):
      if iteration % 1000 == 0:
          print(f'Time elapsed: {(time.time() - start_time)/60:.2f} min')
          print(f'Iteration {iteration} | Loss {loss.item():.2f}\n\n')
          print(evaluate(model, 'Th', 200), '\n')
          print(50*'=')
Time elapsed: 0.00 min
Iteration 0 | Loss 4.58


Th4izvh?=lw2ZaCV_}xEt5y.gA^+r[email protected]<$.1KRQe/c\	{a5A55Dun}_*czf.o6Hmy$l"[email protected]fi{7rjKsvnEMJ
mr`PaKygiE+VSbR#RF|SC^g^CZK,aenDc)t.O_
D^(M]1w'^Wd_HDws\>_2)iavp?*c-npOvoQE>i L 

==================================================
Time elapsed: 2.63 min
Iteration 1000 | Loss 1.81


Th Prost
into
he forn a wock, abrould with his lother the star a caide with the Jue turnd face. Breaknay when and and of or, street were work
have the long is on the proseing bove wabres. Throk a mean h 

==================================================
Time elapsed: 5.29 min
Iteration 2000 | Loss 1.72


Ther face. And civery ire head shook the lange's was note my booked she cray. The grance for that the with Lerary swere were, and for young to-the
wank the tanger brother whas at a for the requestone-st 

==================================================
Time elapsed: 7.91 min
Iteration 3000 | Loss 1.73


Thou my menal known a purntatieful a might
Frent fargefuch by sour that reforned after as as a mists
and the countice of the Founk



"I among him your for the you glason in?"

"I constrance yhabuing a  

==================================================
Time elapsed: 10.55 min
Iteration 4000 | Loss 1.77


The seeantelition pricomer; I have
had the passess bestious had be patriender one up thow, such the even the line
and that ins show was somen of his openey, but fine had a raghter?

"I! And at a sifulra 

==================================================
Time elapsed: 13.17 min
Iteration 5000 | Loss 1.46


The Bask tree. "The intame!"

"Neothing and fam and if you brow lisert, to the mouther desk to an to the Gells that immered of the
indence an aftionation bud, undering to went remark down off work; doe! 

==================================================
Time elapsed: 15.83 min
Iteration 6000 | Loss 1.64


The Pross. What
of moon, and worth her knitting, and is he see myself the was seeper on prisoner her been on him our, and
yet in the poors; is stooness of a morned this things more, were benthell name,  

==================================================
Time elapsed: 18.47 min
Iteration 7000 | Loss 1.64


The here an the ferty care it was
of the streach. As Miss Pross Borring of her surfounds of comprelegual
saken which his returnes, shall in Heaved the arrows
of the retore, then for Defarge. Jark, he wa 

==================================================
Time elapsed: 21.12 min
Iteration 8000 | Loss 1.66


Thur and the decients than any.

Monsiever such put her cite out over the cermarded and in herce then the repariey who
grance the stalled be of the own and conversicted way of his anterom
cold the cirse 

==================================================
Time elapsed: 23.77 min
Iteration 9000 | Loss 1.53


Thrat to his man that extenss of the said her and had and world at it, she had was
as breat--how had asseet triatile of the pationed, and
that worked he works of one and nobainly, and out of that at the 

==================================================
Time elapsed: 26.43 min
Iteration 10000 | Loss 1.30


Ther his moth wooten a new blood, a sile, the lactriden
nother were noter, who had from his father to prettorers his
fation. Then. He are is him a sloke it soits in him woired to the paper women, maning 

==================================================
Time elapsed: 29.07 min
Iteration 11000 | Loss 1.73


The eighs while Miss Pross was
saying a could they last the done by, and pressed to
the been hackeful hight in mending, and to the done into-raid to
have little faming shall now, with the said to go of  

==================================================
Time elapsed: 31.72 min
Iteration 12000 | Loss 1.68


The here. It were would done.

"It alread, I was say?" seen in not in the culles the sunded Miss, sure to be there were the would close
he dark see radfe taken it is instend me had done-all I spy so
str 

==================================================
Time elapsed: 34.35 min
Iteration 13000 | Loss 1.42


The part, but, at had were tosen in
it are of a proined serverently passing the fars, and the
friended that a fiffer the knouttial backle and day, list, and from to could my
deting; and very dark of the 

==================================================
Time elapsed: 37.00 min
Iteration 14000 | Loss 1.78


Thre it was a days.

"You and deperianned of the moved there way, and a socions of the proppiouse and this must a dively?

"Yest!" (And in the care befon this there," asked nother, to the two in this ex 

==================================================
Time elapsed: 39.63 min
Iteration 15000 | Loss 1.54


Tho a man in all looking
the mannen were trangs; he more at man and in the had believe the sick of their an
than the man the prioned in a golderate scattered no stup, and look, all thoused shall
law sca 

==================================================
Time elapsed: 42.28 min
Iteration 16000 | Loss 1.52


Thrishe forth, his have like him
and words of it is a peeched in the eyes farge what
it went exciect the deing and the mittions.

The mounged the repalling's citines of mineurmt you not thinks,
 Charlee 

==================================================
Time elapsed: 44.93 min
Iteration 17000 | Loss 1.58


Thrithest the prisonened I
staid be, short, and not morright door with with the mitting to my worthud no paid
it."

"He do I as a more through a passed and go more.

No and me, the far bold to fears and 

==================================================
Time elapsed: 47.56 min
Iteration 18000 | Loss 1.49


Tho
you would fro in his intides rather sation and chocal went in the things, asked the have hand of the
distened did of the cately roar chifulures. What the His of a not his have pourty
the took this l 

==================================================
Time elapsed: 50.19 min
Iteration 19000 | Loss 1.78


Thragges," said some of
a puncher in the Gabody old, was a Fants tall to know of the complight--seat
more inten asse interancame my any went med Courable hands in that he behing make no will never see t 

==================================================
In [ ]: