This notebook implements linear regression using gradient descent as taught in Week 1 of Coursera's Machine Learning course.
The course doesn't explicitly go into the implementation yet, so I'm not sure how to pick a good alpha (learning rate) nor do I know a good way to determine the number of iterations the algorithm should run.
I decided to attempt a quick implementation in Python anyway in order to improve my understanding.
import pandas as pd
import numpy as np
df = pd.DataFrame({'x': [0, 1, 2, 3, 4, 5], 'y': [1, 3, 5, 7, 9, 11]})
df
x | y | |
---|---|---|
0 | 0 | 1 |
1 | 1 | 3 |
2 | 2 | 5 |
3 | 3 | 7 |
4 | 4 | 9 |
5 | 5 | 11 |
def hypothesis(theta0, theta1, x):
return theta0 + theta1 * x
def linear_regression(df, alpha, iterations):
theta0 = 0
theta1 = 0
m = len(df)
for _ in range(0, iterations):
newTheta0 = 0
newTheta1 = 0
for i in range (0, m):
error = hypothesis(theta0, theta1, df.x[i]) - df.y[i]
newTheta0 = newTheta0 + error
newTheta1 = newTheta1 + error * df.x[i]
newTheta0 = theta0 - alpha / m * newTheta0
newTheta1 = theta1 - alpha / m * newTheta1
theta0 = newTheta0
theta1 = newTheta1
return [theta0, theta1]
%%time
linear_regression(df, 0.1, 1000)
CPU times: user 212 ms, sys: 3.03 ms, total: 215 ms Wall time: 213 ms
[0.99999999999996192, 2.0000000000000107]
The implementation above can be made more efficient by replacing the inner loop with matrix operations instead.
Below is my attempt at doing so:
def vector_linear_regression(df, alpha, iterations):
m = len(df)
thetas = np.zeros((2,1))
X = pd.DataFrame(data={0: 1, 1: df.x}).as_matrix() # Is there a more elegant way to build this matrix?
y = np.array([df.y]).transpose()
op = X.transpose()
for _ in range(0, iterations):
errors = X.dot(thetas) - y
sums = op.dot(errors)
thetas = thetas - sums * alpha / m
return [thetas[0][0], thetas[1][0]]
%%time
vector_linear_regression(df, 0.1, 1000)
CPU times: user 66.5 ms, sys: 3.47 ms, total: 69.9 ms Wall time: 69.2 ms
[0.99999999999996181, 2.0000000000000107]
df.as_matrix()
array([[ 0, 1], [ 1, 3], [ 2, 5], [ 3, 7], [ 4, 9], [ 5, 11]])