import pandas as pd
pd.set_option('display.mpl_style', 'default')
figsize(15,3)
By the end of this chapter, we're going to have downloaded all of Canada's weather data for 2012, and saved it to a CSV.
We'll do this by downloading it one month at a time, and then combining all the months together.
Here's the temperature every hour for 2012!
weather_2012_final = pd.read_csv('../data/weather_2012.csv', index_col='Date/Time')
weather_2012_final['Temp (C)'].plot(figsize=(15, 6))
<matplotlib.axes.AxesSubplot at 0x345b5d0>
When playing with the cycling data, I wanted temperature and precipitation data to find out of people like biking when it's raining. So I went to the site for Canadian historical weather data, and figured out how to get it automatically.
Here we're going to get the data for March 2012, and clean it up
Here's an URL template you can use to get data in Montreal.
url_template = "http://climate.weather.gc.ca/climateData/bulkdata_e.html?format=csv&stationID=5415&Year={year}&Month={month}&timeframe=1&submit=Download+Data"
To get the data for March 2013, we need to format it with month=3, year=2012
.
url = url_template.format(month=3, year=2012)
weather_mar2012 = pd.read_csv(url, skiprows=16, index_col='Date/Time', parse_dates=True, encoding='latin1')
This is super great! We can just use the same read_csv
function as before, and just give it a URL as a filename. Awesome.
There are 16 rows of metadata at the top of this CSV, but pandas knows CSVs are weird, so there's a skiprows
options. We parse the dates again, and set 'Date/Time' to be the index column. Here's the resulting dataframe.
weather_mar2012
<class 'pandas.core.frame.DataFrame'> DatetimeIndex: 744 entries, 2012-03-01 00:00:00 to 2012-03-31 23:00:00 Data columns (total 24 columns): Year 744 non-null values Month 744 non-null values Day 744 non-null values Time 744 non-null values Data Quality 744 non-null values Temp (°C) 744 non-null values Temp Flag 0 non-null values Dew Point Temp (°C) 744 non-null values Dew Point Temp Flag 0 non-null values Rel Hum (%) 744 non-null values Rel Hum Flag 0 non-null values Wind Dir (10s deg) 715 non-null values Wind Dir Flag 0 non-null values Wind Spd (km/h) 744 non-null values Wind Spd Flag 3 non-null values Visibility (km) 744 non-null values Visibility Flag 0 non-null values Stn Press (kPa) 744 non-null values Stn Press Flag 0 non-null values Hmdx 12 non-null values Hmdx Flag 0 non-null values Wind Chill 242 non-null values Wind Chill Flag 1 non-null values Weather 744 non-null values dtypes: float64(14), int64(5), object(5)
Let's plot it!
weather_mar2012[u"Temp (\xb0C)"].plot(figsize=(15, 5))
<matplotlib.axes.AxesSubplot at 0x34e8990>
Notice how it goes up to 25° C in the middle there? That was a big deal. It was March, and people were wearing shorts outside.
And I was out of town and I missed it. Still sad, humans.
I had to write '\xb0'
for that degree character °. Let's get rid of that, to make it easier to type.
weather_mar2012.columns = [s.replace(u'\xb0', '') for s in weather_mar2012.columns]
You'll notice in the summary above that there are a few columns which are are either entirely empty or only have a few values in them. Let's get rid of all of those with dropna
.
The argument axis=1
to dropna
means "drop columns", not rows", and how='any'
means "drop the column if any value is null".
This is much better now -- we only have columns with real data.
weather_mar2012 = weather_mar2012.dropna(axis=1, how='any')
weather_mar2012[:5]
Year | Month | Day | Time | Data Quality | Temp (C) | Dew Point Temp (C) | Rel Hum (%) | Wind Spd (km/h) | Visibility (km) | Stn Press (kPa) | Weather | |
---|---|---|---|---|---|---|---|---|---|---|---|---|
Date/Time | ||||||||||||
2012-03-01 00:00:00 | 2012 | 3 | 1 | 00:00 | -5.5 | -9.7 | 72 | 24 | 4.0 | 100.97 | Snow | |
2012-03-01 01:00:00 | 2012 | 3 | 1 | 01:00 | -5.7 | -8.7 | 79 | 26 | 2.4 | 100.87 | Snow | |
2012-03-01 02:00:00 | 2012 | 3 | 1 | 02:00 | -5.4 | -8.3 | 80 | 28 | 4.8 | 100.80 | Snow | |
2012-03-01 03:00:00 | 2012 | 3 | 1 | 03:00 | -4.7 | -7.7 | 79 | 28 | 4.0 | 100.69 | Snow | |
2012-03-01 04:00:00 | 2012 | 3 | 1 | 04:00 | -5.4 | -7.8 | 83 | 35 | 1.6 | 100.62 | Snow |
The Year/Month/Day/Time columns are redundant, though, and the Data Quality column doesn't look too useful. Let's get rid of those.
The axis=1
argument means "Drop columns", like before. The default for operations like dropna
and drop
is always to operate on rows.
weather_mar2012 = weather_mar2012.drop(['Year', 'Month', 'Day', 'Time', 'Data Quality'], axis=1)
weather_mar2012[:5]
Temp (C) | Dew Point Temp (C) | Rel Hum (%) | Wind Spd (km/h) | Visibility (km) | Stn Press (kPa) | Weather | |
---|---|---|---|---|---|---|---|
Date/Time | |||||||
2012-03-01 00:00:00 | -5.5 | -9.7 | 72 | 24 | 4.0 | 100.97 | Snow |
2012-03-01 01:00:00 | -5.7 | -8.7 | 79 | 26 | 2.4 | 100.87 | Snow |
2012-03-01 02:00:00 | -5.4 | -8.3 | 80 | 28 | 4.8 | 100.80 | Snow |
2012-03-01 03:00:00 | -4.7 | -7.7 | 79 | 28 | 4.0 | 100.69 | Snow |
2012-03-01 04:00:00 | -5.4 | -7.8 | 83 | 35 | 1.6 | 100.62 | Snow |
Awesome! We now only have the relevant columns, and it's much more manageable.
This one's just for fun -- we've already done this before, using groupby and aggregate! We will learn whether or not it gets colder at night. Well, obviously. But let's do it anyway.
temperatures = weather_mar2012[[u'Temp (C)']]
temperatures['Hour'] = weather_mar2012.index.hour
temperatures.groupby('Hour').aggregate(np.median).plot()
<matplotlib.axes.AxesSubplot at 0x34ec610>
So it looks like the time with the highest median temperature is 2pm. Neat.
Okay, so what if we want the data for the whole year? Ideally the API would just let us download that, but I couldn't figure out a way to do that.
First, let's put our work from above into a function that gets the weather for a given month.
I noticed that there's an irritating bug where when I ask for January, it gives me data for the previous year, so we'll fix that too. [no, really. You can check =)]
def download_weather_month(year, month):
if month == 1:
year += 1
url = url_template.format(year=year, month=month)
weather_data = pd.read_csv(url, skiprows=16, index_col='Date/Time', parse_dates=True)
weather_data = weather_data.dropna(axis=1)
weather_data.columns = [col.replace('\xb0', '') for col in weather_data.columns]
weather_data = weather_data.drop(['Year', 'Day', 'Month', 'Time', 'Data Quality'], axis=1)
return weather_data
We can test that this function does the right thing:
download_weather_month(2012, 1)[:5]
Temp (C) | Dew Point Temp (C) | Rel Hum (%) | Wind Spd (km/h) | Visibility (km) | Stn Press (kPa) | Weather | |
---|---|---|---|---|---|---|---|
Date/Time | |||||||
2012-01-01 00:00:00 | -1.8 | -3.9 | 86 | 4 | 8.0 | 101.24 | Fog |
2012-01-01 01:00:00 | -1.8 | -3.7 | 87 | 4 | 8.0 | 101.24 | Fog |
2012-01-01 02:00:00 | -1.8 | -3.4 | 89 | 7 | 4.0 | 101.26 | Freezing Drizzle,Fog |
2012-01-01 03:00:00 | -1.5 | -3.2 | 88 | 6 | 4.0 | 101.27 | Freezing Drizzle,Fog |
2012-01-01 04:00:00 | -1.5 | -3.3 | 88 | 7 | 4.8 | 101.23 | Fog |
Now we can get all the months at once. This will take a little while to run.
data_by_month = [download_weather_month(2012, i) for i in range(1, 13)]
Once we have this, it's easy to concatenate all the dataframes together into one big dataframe using pd.concat
. And now we have the whole year's data!
weather_2012 = pd.concat(data_by_month)
weather_2012
<class 'pandas.core.frame.DataFrame'> DatetimeIndex: 8784 entries, 2012-01-01 00:00:00 to 2012-12-31 23:00:00 Data columns (total 7 columns): Temp (C) 8784 non-null values Dew Point Temp (C) 8784 non-null values Rel Hum (%) 8784 non-null values Wind Spd (km/h) 8784 non-null values Visibility (km) 8784 non-null values Stn Press (kPa) 8784 non-null values Weather 8784 non-null values dtypes: float64(4), int64(2), object(1)
It's slow and unnecessary to download the data every time, so let's save our dataframe:
weather_2012.to_csv('../data/weather_2012.csv')
And we're done!