This notebook covers a Python-based solution for the first programming exercise of the machine learning class on Coursera. Please refer to the exercise text for detailed descriptions and equations.
In this exercise we'll implement simple linear regression using gradient descent and apply it to an example problem. We'll also extend our implementation to handle multiple variables and apply it to a slightly more difficult example.
In the first part of the exercise, we're tasked with implementing linear regression with one variable to predict profits for a food truck. Suppose you are the CEO of a restaurant franchise and are considering different cities for opening a new outlet. The chain already has trucks in various cities and you have data for profits and populations from the cities.
Let's start by importing some libraries and examining the data.
import numpy as np import pandas as pd import matplotlib.pyplot as plt %matplotlib inline
import os path = os.getcwd() + '\data\ex1data1.txt' data = pd.read_csv(path, header=None, names=['Population', 'Profit']) data.head()
Let's plot it to get a better idea of what the data looks like.
data.plot(kind='scatter', x='Population', y='Profit', figsize=(12,8))
<matplotlib.axes._subplots.AxesSubplot at 0xd140198>
Now let's implement linear regression using gradient descent to minimize the cost function. The equations implemented in the following code samples are detailed in "ex1.pdf" in the "exercises" folder.
First we'll create a function to compute the cost of a given solution (characterized by the parameters theta).
def computeCost(X, y, theta): inner = np.power(((X * theta.T) - y), 2) return np.sum(inner) / (2 * len(X))
Let's add a column of ones to the training set so we can use a vectorized solution to computing the cost and gradients.
data.insert(0, 'Ones', 1)
Now let's do some variable initialization.
# set X (training data) and y (target variable) cols = data.shape X = data.iloc[:,0:cols-1] y = data.iloc[:,cols-1:cols]
Let's take a look to make sure X (training set) and y (target variable) look correct.
The cost function is expecting numpy matrices so we need to convert X and y before we can use them. We also need to initialize theta.
X = np.matrix(X.values) y = np.matrix(y.values) theta = np.matrix(np.array([0,0]))
Here's what theta looks like.
Let's take a quick look at the shape of our matrices.
X.shape, theta.shape, y.shape
((97L, 2L), (1L, 2L), (97L, 1L))
Now let's compute the cost for our initial solution (0 values for theta).
computeCost(X, y, theta)
So far so good. Now we need to define a function to perform gradient descent on the parameters theta using the update rules defined in the text.
def gradientDescent(X, y, theta, alpha, iters): temp = np.matrix(np.zeros(theta.shape)) parameters = int(theta.ravel().shape) cost = np.zeros(iters) for i in range(iters): error = (X * theta.T) - y for j in range(parameters): term = np.multiply(error, X[:,j]) temp[0,j] = theta[0,j] - ((alpha / len(X)) * np.sum(term)) theta = temp cost[i] = computeCost(X, y, theta) return theta, cost
Initialize some additional variables - the learning rate alpha, and the number of iterations to perform.
alpha = 0.01 iters = 1000
Now let's run the gradient descent algorithm to fit our parameters theta to the training set.
g, cost = gradientDescent(X, y, theta, alpha, iters) g
matrix([[-3.24140214, 1.1272942 ]])
Finally we can compute the cost (error) of the trained model using our fitted parameters.
computeCost(X, y, g)
Now let's plot the linear model along with the data to visually see how well it fits.
x = np.linspace(data.Population.min(), data.Population.max(), 100) f = g[0, 0] + (g[0, 1] * x) fig, ax = plt.subplots(figsize=(12,8)) ax.plot(x, f, 'r', label='Prediction') ax.scatter(data.Population, data.Profit, label='Traning Data') ax.legend(loc=2) ax.set_xlabel('Population') ax.set_ylabel('Profit') ax.set_title('Predicted Profit vs. Population Size')
<matplotlib.text.Text at 0xd35a518>
Looks pretty good! Since the gradient decent function also outputs a vector with the cost at each training iteration, we can plot that as well. Notice that the cost always decreases - this is an example of a convex optimization problem.
fig, ax = plt.subplots(figsize=(12,8)) ax.plot(np.arange(iters), cost, 'r') ax.set_xlabel('Iterations') ax.set_ylabel('Cost') ax.set_title('Error vs. Training Epoch')
<matplotlib.text.Text at 0xd5bccc0>
Exercise 1 also included a housing price data set with 2 variables (size of the house in square feet and number of bedrooms) and a target (price of the house). Let's use the techniques we already applied to analyze that data set as well.
path = os.getcwd() + '\data\ex1data2.txt' data2 = pd.read_csv(path, header=None, names=['Size', 'Bedrooms', 'Price']) data2.head()
For this task we add another pre-processing step - normalizing the features. This is very easy with pandas.
data2 = (data2 - data2.mean()) / data2.std() data2.head()
Now let's repeat our pre-processing steps from part 1 and run the linear regression procedure on the new data set.
# add ones column data2.insert(0, 'Ones', 1) # set X (training data) and y (target variable) cols = data2.shape X2 = data2.iloc[:,0:cols-1] y2 = data2.iloc[:,cols-1:cols] # convert to matrices and initialize theta X2 = np.matrix(X2.values) y2 = np.matrix(y2.values) theta2 = np.matrix(np.array([0,0,0])) # perform linear regression on the data set g2, cost2 = gradientDescent(X2, y2, theta2, alpha, iters) # get the cost (error) of the model computeCost(X2, y2, g2)
We can take a quick look at the training progess for this one as well.
fig, ax = plt.subplots(figsize=(12,8)) ax.plot(np.arange(iters), cost2, 'r') ax.set_xlabel('Iterations') ax.set_ylabel('Cost') ax.set_title('Error vs. Training Epoch')
<matplotlib.text.Text at 0xd7bb240>
Instead of implementing these algorithms from scratch, we could also use scikit-learn's linear regression function. Let's apply scikit-learn's linear regressio algorithm to the data from part 1 and see what it comes up with.
from sklearn import linear_model model = linear_model.LinearRegression() model.fit(X, y)
LinearRegression(copy_X=True, fit_intercept=True, normalize=False)
Here's what the scikit-learn model's predictions look like.
x = np.array(X[:, 1].A1) f = model.predict(X).flatten() fig, ax = plt.subplots(figsize=(12,8)) ax.plot(x, f, 'r', label='Prediction') ax.scatter(data.Population, data.Profit, label='Traning Data') ax.legend(loc=2) ax.set_xlabel('Population') ax.set_ylabel('Profit') ax.set_title('Predicted Profit vs. Population Size')
<matplotlib.text.Text at 0xf258860>
That's it! Thanks for reading. In Exercise 2 we'll take a look at logistic regression for classification problems.