Important: This notebook will only work with fastai-0.7.x. Do not try to run any fastai-1.x code from this path in the repository because it will load fastai-0.7.x

Movielens

In [ ]:
%reload_ext autoreload
%autoreload 2
%matplotlib inline

from fastai.learner import *
from fastai.column_data import *
In [ ]:
path='data/ml-latest-small/'

We're working with the movielens data, which contains one rating per row, like this:

In [ ]:
ratings = pd.read_csv(path+'ratings.csv')
ratings.head()
Out[ ]:
userId movieId rating timestamp
0 1 31 2.5 1260759144
1 1 1029 3.0 1260759179
2 1 1061 3.0 1260759182
3 1 1129 2.0 1260759185
4 1 1172 4.0 1260759205

Just for display purposes, let's read in the movie names too.

In [ ]:
movies = pd.read_csv(path+'movies.csv')
movies.head()
Out[ ]:
movieId title genres
0 1 Toy Story (1995) Adventure|Animation|Children|Comedy|Fantasy
1 2 Jumanji (1995) Adventure|Children|Fantasy
2 3 Grumpier Old Men (1995) Comedy|Romance
3 4 Waiting to Exhale (1995) Comedy|Drama|Romance
4 5 Father of the Bride Part II (1995) Comedy

Create subset for Excel

We create a crosstab of the most popular movies and most movie-addicted users which we'll copy into Excel for creating a simple example. This isn't necessary for any of the modeling below however.

In [ ]:
g=ratings.groupby('userId')['rating'].count()
topUsers=g.sort_values(ascending=False)[:15]

g=ratings.groupby('movieId')['rating'].count()
topMovies=g.sort_values(ascending=False)[:15]

top_r = ratings.join(topUsers, rsuffix='_r', how='inner', on='userId')
top_r = top_r.join(topMovies, rsuffix='_r', how='inner', on='movieId')

pd.crosstab(top_r.userId, top_r.movieId, top_r.rating, aggfunc=np.sum)
Out[ ]:
movieId 1 110 260 296 318 356 480 527 589 593 608 1196 1198 1270 2571
userId
15 2.0 3.0 5.0 5.0 2.0 1.0 3.0 4.0 4.0 5.0 5.0 5.0 4.0 5.0 5.0
30 4.0 5.0 4.0 5.0 5.0 5.0 4.0 5.0 4.0 4.0 5.0 4.0 5.0 5.0 3.0
73 5.0 4.0 4.5 5.0 5.0 5.0 4.0 5.0 3.0 4.5 4.0 5.0 5.0 5.0 4.5
212 3.0 5.0 4.0 4.0 4.5 4.0 3.0 5.0 3.0 4.0 NaN NaN 3.0 3.0 5.0
213 3.0 2.5 5.0 NaN NaN 2.0 5.0 NaN 4.0 2.5 2.0 5.0 3.0 3.0 4.0
294 4.0 3.0 4.0 NaN 3.0 4.0 4.0 4.0 3.0 NaN NaN 4.0 4.5 4.0 4.5
311 3.0 3.0 4.0 3.0 4.5 5.0 4.5 5.0 4.5 2.0 4.0 3.0 4.5 4.5 4.0
380 4.0 5.0 4.0 5.0 4.0 5.0 4.0 NaN 4.0 5.0 4.0 4.0 NaN 3.0 5.0
452 3.5 4.0 4.0 5.0 5.0 4.0 5.0 4.0 4.0 5.0 5.0 4.0 4.0 4.0 2.0
468 4.0 3.0 3.5 3.5 3.5 3.0 2.5 NaN NaN 3.0 4.0 3.0 3.5 3.0 3.0
509 3.0 5.0 5.0 5.0 4.0 4.0 3.0 5.0 2.0 4.0 4.5 5.0 5.0 3.0 4.5
547 3.5 NaN NaN 5.0 5.0 2.0 3.0 5.0 NaN 5.0 5.0 2.5 2.0 3.5 3.5
564 4.0 1.0 2.0 5.0 NaN 3.0 5.0 4.0 5.0 5.0 5.0 5.0 5.0 3.0 3.0
580 4.0 4.5 4.0 4.5 4.0 3.5 3.0 4.0 4.5 4.0 4.5 4.0 3.5 3.0 4.5
624 5.0 NaN 5.0 5.0 NaN 3.0 3.0 NaN 3.0 5.0 4.0 5.0 5.0 5.0 2.0

Collaborative filtering

In [ ]:
val_idxs = get_cv_idxs(len(ratings))
wd=2e-4
n_factors = 50
In [ ]:
cf = CollabFilterDataset.from_csv(path, 'ratings.csv', 'userId', 'movieId', 'rating')
learn = cf.get_learner(n_factors, val_idxs, 64, opt_fn=optim.Adam)
In [ ]:
learn.fit(1e-2, 2, wds=wd, cycle_len=1, cycle_mult=2)
[ 0.       0.84581  0.82454]                                    
[ 1.       0.75627  0.93568]                                    
[ 2.       0.29527  0.89037]                                    

Let's compare to some benchmarks. Here's some benchmarks on the same dataset for the popular Librec system for collaborative filtering. They show best results based on RMSE of 0.91. We'll need to take the square root of our loss, since we use plain MSE.

In [ ]:
math.sqrt(0.776)
Out[ ]:
0.8809086218218096

Looking good - we've found a solution better than any of those benchmarks! Let's take a look at how the predictions compare to actuals for this model.

In [ ]:
preds = learn.predict()
In [ ]:
y=learn.data.val_y
sns.jointplot(preds, y, kind='hex', stat_func=None);

Analyze results

Movie bias

In [ ]:
movie_names = movies.set_index('movieId')['title'].to_dict()
g=ratings.groupby('movieId')['rating'].count()
topMovies=g.sort_values(ascending=False).index.values[:3000]
topMovieIdx = np.array([cf.item2idx[o] for o in topMovies])
In [ ]:
m=learn.model; m.cuda()
Out[ ]:
EmbeddingDotBias (
  (u): Embedding(671, 50)
  (i): Embedding(9066, 50)
  (ub): Embedding(671, 1)
  (ib): Embedding(9066, 1)
)

First, we'll look at the movie bias term. Here, our input is the movie id (a single id), and the output is the movie bias (a single float).

In [ ]:
movie_bias = to_np(m.ib(V(topMovieIdx)))
In [ ]:
movie_bias
Out[ ]:
array([[ 0.80916],
       [ 0.85248],
       [ 1.13049],
       ..., 
       [ 0.52353],
       [-0.0782 ],
       [ 0.33268]], dtype=float32)
In [ ]:
movie_ratings = [(b[0], movie_names[i]) for i,b in zip(topMovies,movie_bias)]

Now we can look at the top and bottom rated movies. These ratings are corrected for different levels of reviewer sentiment, as well as different types of movies that different reviewers watch.

In [ ]:
sorted(movie_ratings, key=lambda o: o[0])[:15]
Out[ ]:
[(-0.96070349, 'Battlefield Earth (2000)'),
 (-0.76858485, 'Speed 2: Cruise Control (1997)'),
 (-0.73675376, 'Wild Wild West (1999)'),
 (-0.73655486, 'Anaconda (1997)'),
 (-0.72457194, 'Super Mario Bros. (1993)'),
 (-0.69564718, 'Congo (1995)'),
 (-0.67043746, 'Superman III (1983)'),
 (-0.64385736, 'Mighty Morphin Power Rangers: The Movie (1995)'),
 (-0.62750411, "Joe's Apartment (1996)"),
 (-0.60154277, 'Police Academy 4: Citizens on Patrol (1987)'),
 (-0.59929478, 'Batman & Robin (1997)'),
 (-0.59667748, 'Jaws 3-D (1983)'),
 (-0.5921765, 'Dungeons & Dragons (2000)'),
 (-0.59074384, 'Inspector Gadget (1999)'),
 (-0.57559621, 'Haunting, The (1999)')]
In [ ]:
sorted(movie_ratings, key=itemgetter(0))[:15]
Out[ ]:
[(-0.96070349, 'Battlefield Earth (2000)'),
 (-0.76858485, 'Speed 2: Cruise Control (1997)'),
 (-0.73675376, 'Wild Wild West (1999)'),
 (-0.73655486, 'Anaconda (1997)'),
 (-0.72457194, 'Super Mario Bros. (1993)'),
 (-0.69564718, 'Congo (1995)'),
 (-0.67043746, 'Superman III (1983)'),
 (-0.64385736, 'Mighty Morphin Power Rangers: The Movie (1995)'),
 (-0.62750411, "Joe's Apartment (1996)"),
 (-0.60154277, 'Police Academy 4: Citizens on Patrol (1987)'),
 (-0.59929478, 'Batman & Robin (1997)'),
 (-0.59667748, 'Jaws 3-D (1983)'),
 (-0.5921765, 'Dungeons & Dragons (2000)'),
 (-0.59074384, 'Inspector Gadget (1999)'),
 (-0.57559621, 'Haunting, The (1999)')]
In [ ]:
sorted(movie_ratings, key=lambda o: o[0], reverse=True)[:15]
Out[ ]:
[(1.3070084, 'Shawshank Redemption, The (1994)'),
 (1.1196285, 'Godfather, The (1972)'),
 (1.0844109, 'Usual Suspects, The (1995)'),
 (0.96578616, "Schindler's List (1993)"),
 (0.90921378, 'Silence of the Lambs, The (1991)'),
 (0.89407367, 'Godfather: Part II, The (1974)'),
 (0.87860429, '12 Angry Men (1957)'),
 (0.87099487, 'Pulp Fiction (1994)'),
 (0.85731125, 'Memento (2000)'),
 (0.85192037, 'Matrix, The (1999)'),
 (0.84797066, 'Dark Knight, The (2008)'),
 (0.8479442, 'To Kill a Mockingbird (1962)'),
 (0.83670187, 'Forrest Gump (1994)'),
 (0.8192088, 'Star Wars: Episode V - The Empire Strikes Back (1980)'),
 (0.81887919, 'Rear Window (1954)')]

Embedding interpretation

We can now do the same thing for the embeddings.

In [ ]:
movie_emb = to_np(m.i(V(topMovieIdx)))
movie_emb.shape
Out[ ]:
(3000, 50)

Because it's hard to interpret 50 embeddings, we use PCA to simplify them down to just 3 vectors.

In [ ]:
from sklearn.decomposition import PCA
pca = PCA(n_components=3)
movie_pca = pca.fit(movie_emb.T).components_
In [ ]:
movie_pca.shape
Out[ ]:
(3, 3000)
In [ ]:
fac0 = movie_pca[0]
movie_comp = [(f, movie_names[i]) for f,i in zip(fac0, topMovies)]

Here's the 1st component. It seems to be 'easy watching' vs 'serious'.

In [ ]:
sorted(movie_comp, key=itemgetter(0), reverse=True)[:10]
Out[ ]:
[(0.06748189, 'Independence Day (a.k.a. ID4) (1996)'),
 (0.061572548, 'Police Academy 4: Citizens on Patrol (1987)'),
 (0.061050549, 'Waterworld (1995)'),
 (0.057877172, 'Rocky V (1990)'),
 (0.057183612, 'Home Alone 3 (1997)'),
 (0.056849808, 'Armageddon (1998)'),
 (0.056735475, 'Miss Congeniality (2000)'),
 (0.054530937, 'Outbreak (1995)'),
 (0.053475372, 'Evolution (2001)'),
 (0.052995622, 'Pay It Forward (2000)')]
In [ ]:
sorted(movie_comp, key=itemgetter(0))[:10]
Out[ ]:
[(-0.078433245, 'Godfather: Part II, The (1974)'),
 (-0.072180331, 'Fargo (1996)'),
 (-0.071351372, 'Pulp Fiction (1994)'),
 (-0.068537779, 'Goodfellas (1990)'),
 (-0.067418814, 'Chinatown (1974)'),
 (-0.066787124, 'Taxi Driver (1976)'),
 (-0.06392362, 'Apocalypse Now (1979)'),
 (-0.060093477, 'Brokeback Mountain (2005)'),
 (-0.057078246, 'Godfather, The (1972)'),
 (-0.055729419, 'Player, The (1992)')]
In [ ]:
fac1 = movie_pca[1]
movie_comp = [(f, movie_names[i]) for f,i in zip(fac1, topMovies)]

Here's the 2nd component. It seems to be 'CGI' vs 'dialog driven'.

In [ ]:
sorted(movie_comp, key=itemgetter(0), reverse=True)[:10]
Out[ ]:
[(0.058975246, 'Bonfire of the Vanities (1990)'),
 (0.055992026, '2001: A Space Odyssey (1968)'),
 (0.054682467, 'Tank Girl (1995)'),
 (0.054429606, 'Purple Rose of Cairo, The (1985)'),
 (0.050998077, 'Mulholland Drive (2001)'),
 (0.049576689, "Joe's Apartment (1996)"),
 (0.047549088, 'What Ever Happened to Baby Jane? (1962)'),
 (0.046446536, 'Island of Dr. Moreau, The (1996)'),
 (0.045140576, 'L.A. Story (1991)'),
 (0.045048587, 'Mouse Hunt (1997)')]
In [ ]:
sorted(movie_comp, key=itemgetter(0))[:10]
Out[ ]:
[(-0.1064609, 'Lord of the Rings: The Return of the King, The (2003)'),
 (-0.090635143, 'Aladdin (1992)'),
 (-0.089208141, 'Star Wars: Episode V - The Empire Strikes Back (1980)'),
 (-0.088854566, 'Star Wars: Episode IV - A New Hope (1977)'),
 (-0.085997969, 'Beauty and the Beast (1991)'),
 (-0.085541978, "Schindler's List (1993)"),
 (-0.080922142, 'Saving Private Ryan (1998)'),
 (-0.079378694,
  'Raiders of the Lost Ark (Indiana Jones and the Raiders of the Lost Ark) (1981)'),
 (-0.079295151, 'Lord of the Rings: The Two Towers, The (2002)'),
 (-0.078875825, 'My Big Fat Greek Wedding (2002)')]

We can draw a picture to see how various movies appear on the map of these components. This picture shows the first two components.

In [ ]:
idxs = np.random.choice(len(topMovies), 50, replace=False)
X = fac0[idxs]
Y = fac1[idxs]
plt.figure(figsize=(15,15))
plt.scatter(X, Y)
for i, x, y in zip(topMovies[idxs], X, Y):
    plt.text(x,y,movie_names[i], color=np.random.rand(3)*0.7, fontsize=11)
plt.show()

Collab filtering from scratch

Dot product example

In [ ]:
a = T([[1.,2],[3,4]])
b = T([[2.,2],[10,10]])
a,b
Out[ ]:
(
  1  2
  3  4
 [torch.FloatTensor of size 2x2], 
   2   2
  10  10
 [torch.FloatTensor of size 2x2])
In [ ]:
a*b
Out[ ]:
  2   4
 30  40
[torch.FloatTensor of size 2x2]
In [ ]:
(a*b).sum(1)
Out[ ]:
  6
 70
[torch.FloatTensor of size 2]
In [ ]:
class DotProduct(nn.Module):
    def forward(self, u, m): return (u*m).sum(1)
In [ ]:
model=DotProduct()
In [ ]:
model(a,b)
Out[ ]:
  6
 70
[torch.FloatTensor of size 2]

Dot product model

In [ ]:
u_uniq = ratings.userId.unique()
user2idx = {o:i for i,o in enumerate(u_uniq)}
ratings.userId = ratings.userId.apply(lambda x: user2idx[x])

m_uniq = ratings.movieId.unique()
movie2idx = {o:i for i,o in enumerate(m_uniq)}
ratings.movieId = ratings.movieId.apply(lambda x: movie2idx[x])

n_users=int(ratings.userId.nunique())
n_movies=int(ratings.movieId.nunique())
In [ ]:
class EmbeddingDot(nn.Module):
    def __init__(self, n_users, n_movies):
        super().__init__()
        self.u = nn.Embedding(n_users, n_factors)
        self.m = nn.Embedding(n_movies, n_factors)
        self.u.weight.data.uniform_(0,0.05)
        self.m.weight.data.uniform_(0,0.05)
        
    def forward(self, cats, conts):
        users,movies = cats[:,0],cats[:,1]
        u,m = self.u(users),self.m(movies)
        return (u*m).sum(1).view(-1, 1)
In [ ]:
x = ratings.drop(['rating', 'timestamp'],axis=1)
y = ratings['rating'].astype(np.float32)
In [ ]:
data = ColumnarModelData.from_data_frame(path, val_idxs, x, y, ['userId', 'movieId'], 64)
In [ ]:
wd=1e-5
model = EmbeddingDot(n_users, n_movies).cuda()
opt = optim.SGD(model.parameters(), 1e-1, weight_decay=wd, momentum=0.9)
In [ ]:
fit(model, data, 3, opt, F.mse_loss)
[ 0.      1.6368  1.6415]                                      
[ 1.       1.13749  1.29373]                                   
[ 2.       0.89736  1.22818]                                    

In [ ]:
set_lrs(opt, 0.01)
In [ ]:
fit(model, data, 3, opt, F.mse_loss)
[ 0.       0.69763  1.14979]                                    
[ 1.       0.70115  1.13657]                                    
[ 2.       0.66739  1.1303 ]                                    

Bias

In [ ]:
min_rating,max_rating = ratings.rating.min(),ratings.rating.max()
min_rating,max_rating
Out[ ]:
(0.5, 5.0)
In [ ]:
def get_emb(ni,nf):
    e = nn.Embedding(ni, nf)
    e.weight.data.uniform_(-0.01,0.01)
    return e

class EmbeddingDotBias(nn.Module):
    def __init__(self, n_users, n_movies):
        super().__init__()
        (self.u, self.m, self.ub, self.mb) = [get_emb(*o) for o in [
            (n_users, n_factors), (n_movies, n_factors), (n_users,1), (n_movies,1)
        ]]
        
    def forward(self, cats, conts):
        users,movies = cats[:,0],cats[:,1]
        um = (self.u(users)* self.m(movies)).sum(1)
        res = um + self.ub(users).squeeze() + self.mb(movies).squeeze()
        res = F.sigmoid(res) * (max_rating-min_rating) + min_rating
        return res.view(-1, 1)
In [ ]:
wd=2e-4
model = EmbeddingDotBias(cf.n_users, cf.n_items).cuda()
opt = optim.SGD(model.parameters(), 1e-1, weight_decay=wd, momentum=0.9)
In [ ]:
fit(model, data, 3, opt, F.mse_loss)
[ 0.       0.88212  0.83626]                                    
[ 1.       0.8108   0.81831]                                    
[ 2.       0.78864  0.80989]                                    

In [ ]:
set_lrs(opt, 1e-2)
In [ ]:
fit(model, data, 3, opt, F.mse_loss)
[ 0.       0.72795  0.80337]                                    
[ 1.       0.75064  0.80203]                                    
[ 2.       0.75122  0.80124]                                    

Mini net

In [ ]:
class EmbeddingNet(nn.Module):
    def __init__(self, n_users, n_movies, nh=10, p1=0.05, p2=0.5):
        super().__init__()
        (self.u, self.m) = [get_emb(*o) for o in [
            (n_users, n_factors), (n_movies, n_factors)]]
        self.lin1 = nn.Linear(n_factors*2, nh)
        self.lin2 = nn.Linear(nh, 1)
        self.drop1 = nn.Dropout(p1)
        self.drop2 = nn.Dropout(p2)
        
    def forward(self, cats, conts):
        users,movies = cats[:,0],cats[:,1]
        x = self.drop1(torch.cat([self.u(users),self.m(movies)], dim=1))
        x = self.drop2(F.relu(self.lin1(x)))
        return F.sigmoid(self.lin2(x)) * (max_rating-min_rating+1) + min_rating-0.5
In [ ]:
wd=1e-5
model = EmbeddingNet(n_users, n_movies).cuda()
opt = optim.Adam(model.parameters(), 1e-3, weight_decay=wd)
In [ ]:
fit(model, data, 3, opt, F.mse_loss)
[ 0.       0.88043  0.82363]                                    
[ 1.       0.8941   0.81264]                                    
[ 2.       0.86179  0.80706]                                    

In [ ]:
set_lrs(opt, 1e-3)
In [ ]:
fit(model, data, 3, opt, F.mse_loss)
[ 0.       0.7669   0.78622]                                    
[ 1.       0.74277  0.78152]                                    
[ 2.       0.69891  0.78075]