# Lineare Regression¶

This notebook implements linear regression using gradient descent as taught in Week 1 of Coursera's Machine Learning course.

The course doesn't explicitly go into the implementation yet, so I'm not sure how to pick a good alpha (learning rate) nor do I know a good way to determine the number of iterations the algorithm should run.

I decided to attempt a quick implementation in Python anyway in order to improve my understanding.

In [1]:
import pandas as pd
import numpy as np

In [2]:
df = pd.DataFrame({'x': [0, 1, 2, 3, 4, 5], 'y': [1, 3, 5, 7, 9, 11]})
df

Out[2]:
x y
0 0 1
1 1 3
2 2 5
3 3 7
4 4 9
5 5 11
In [3]:
def hypothesis(theta0, theta1, x):
return theta0 + theta1 * x

def linear_regression(df, alpha, iterations):
theta0 = 0
theta1 = 0
m = len(df)
for _ in range(0, iterations):
newTheta0 = 0
newTheta1 = 0
for i in range (0, m):
error = hypothesis(theta0, theta1, df.x[i]) - df.y[i]
newTheta0 = newTheta0 + error
newTheta1 = newTheta1 + error * df.x[i]
newTheta0 = theta0 - alpha / m * newTheta0
newTheta1 = theta1 - alpha / m * newTheta1
theta0 = newTheta0
theta1 = newTheta1
return [theta0, theta1]

In [4]:
%%time
linear_regression(df, 0.1, 1000)

CPU times: user 212 ms, sys: 3.03 ms, total: 215 ms
Wall time: 213 ms

Out[4]:
[0.99999999999996192, 2.0000000000000107]

The implementation above can be made more efficient by replacing the inner loop with matrix operations instead.

Below is my attempt at doing so:

In [5]:
def vector_linear_regression(df, alpha, iterations):
m = len(df)
thetas = np.zeros((2,1))
X = pd.DataFrame(data={0: 1, 1: df.x}).as_matrix() # Is there a more elegant way to build this matrix?
y = np.array([df.y]).transpose()
op = X.transpose()
for _ in range(0, iterations):
errors = X.dot(thetas) - y
sums = op.dot(errors)
thetas = thetas - sums * alpha / m
return [thetas[0][0], thetas[1][0]]

In [6]:
%%time
vector_linear_regression(df, 0.1, 1000)

CPU times: user 66.5 ms, sys: 3.47 ms, total: 69.9 ms
Wall time: 69.2 ms

Out[6]:
[0.99999999999996181, 2.0000000000000107]
In [7]:
df.as_matrix()

Out[7]:
array([[ 0,  1],
[ 1,  3],
[ 2,  5],
[ 3,  7],
[ 4,  9],
[ 5, 11]])