alt text

A Deep Dive into Geospatial Analysis

Many of the datasets that data scientists handle have some kind of geospatial component to them, and that information is oftentimes useful-to-critical for understanding the problem at hand. As such, an understanding of spatial data and how to work with it is a valuable skill for any data scientist to have. Even better, Python provides a rich toolset for working in this domain, and recent advances have greatly simplified and consolidated these.

In this tutorial we will take a deep dive into geospatial analysis in Python, using tools like geopandas, shapely, and pysal to analyze a dataset, provided by Kaggle (and originally from Inside AirBnB), of sample AirBnB locations in Boston, Massachusetts.

This tutorial is targeted at folks who know a thing or two about data but haven't used Python's geospatial data tools just yet. As such, it assumes a high level of familiarity with pandas. Some familiarity with scikit-learn, statsmodels, matplotlib, and seaborn is also helpful.

In [1]:
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
pd.set_option("max_columns", None)

Plotting Points

First, let's boot up and examine our data. Since our data comes in a simple CSV file, we load it into a pandas DataFrame.

In [353]:
listings = pd.read_csv("../input/listings.csv")
In [4]:
listings.head()
Out[4]:
id listing_url scrape_id last_scraped name summary space description experiences_offered neighborhood_overview notes transit access interaction house_rules thumbnail_url medium_url picture_url xl_picture_url host_id host_url host_name host_since host_location host_about host_response_time host_response_rate host_acceptance_rate host_is_superhost host_thumbnail_url host_picture_url host_neighbourhood host_listings_count host_total_listings_count host_verifications host_has_profile_pic host_identity_verified street neighbourhood neighbourhood_cleansed neighbourhood_group_cleansed city state zipcode market smart_location country_code country latitude longitude is_location_exact property_type room_type accommodates bathrooms bedrooms beds bed_type amenities square_feet price weekly_price monthly_price security_deposit cleaning_fee guests_included extra_people minimum_nights maximum_nights calendar_updated has_availability availability_30 availability_60 availability_90 availability_365 calendar_last_scraped number_of_reviews first_review last_review review_scores_rating review_scores_accuracy review_scores_cleanliness review_scores_checkin review_scores_communication review_scores_location review_scores_value requires_license license jurisdiction_names instant_bookable cancellation_policy require_guest_profile_picture require_guest_phone_verification calculated_host_listings_count reviews_per_month
0 12147973 https://www.airbnb.com/rooms/12147973 20160906204935 2016-09-07 Sunny Bungalow in the City Cozy, sunny, family home. Master bedroom high... The house has an open and cozy feel at the sam... Cozy, sunny, family home. Master bedroom high... none Roslindale is quiet, convenient and friendly. ... NaN The bus stop is 2 blocks away, and frequent. B... You will have access to 2 bedrooms, a living r... NaN Clean up and treat the home the way you'd like... https://a2.muscache.com/im/pictures/c0842db1-e... https://a2.muscache.com/im/pictures/c0842db1-e... https://a2.muscache.com/im/pictures/c0842db1-e... https://a2.muscache.com/im/pictures/c0842db1-e... 31303940 https://www.airbnb.com/users/show/31303940 Virginia 2015-04-15 Boston, Massachusetts, United States We are country and city connecting in our deck... NaN NaN NaN f https://a2.muscache.com/im/pictures/5936fef0-b... https://a2.muscache.com/im/pictures/5936fef0-b... Roslindale 1 1 ['email', 'phone', 'facebook', 'reviews'] t f Birch Street, Boston, MA 02131, United States Roslindale Roslindale NaN Boston MA 02131 Boston Boston, MA US United States 42.282619 -71.133068 t House Entire home/apt 4 1.5 2.0 3.0 Real Bed {TV,"Wireless Internet",Kitchen,"Free Parking ... NaN $250.00 NaN NaN NaN $35.00 1 $0.00 2 1125 2 weeks ago NaN 0 0 0 0 2016-09-06 0 NaN NaN NaN NaN NaN NaN NaN NaN NaN f NaN NaN f moderate f f 1 NaN
1 3075044 https://www.airbnb.com/rooms/3075044 20160906204935 2016-09-07 Charming room in pet friendly apt Charming and quiet room in a second floor 1910... Small but cozy and quite room with a full size... Charming and quiet room in a second floor 1910... none The room is in Roslindale, a diverse and prima... If you don't have a US cell phone, you can tex... Plenty of safe street parking. Bus stops a few... Apt has one more bedroom (which I use) and lar... If I am at home, I am likely working in my hom... Pet friendly but please confirm with me if the... https://a1.muscache.com/im/pictures/39327812/d... https://a1.muscache.com/im/pictures/39327812/d... https://a1.muscache.com/im/pictures/39327812/d... https://a1.muscache.com/im/pictures/39327812/d... 2572247 https://www.airbnb.com/users/show/2572247 Andrea 2012-06-07 Boston, Massachusetts, United States I live in Boston and I like to travel and have... within an hour 100% 100% f https://a2.muscache.com/im/users/2572247/profi... https://a2.muscache.com/im/users/2572247/profi... Roslindale 1 1 ['email', 'phone', 'facebook', 'linkedin', 'am... t t Pinehurst Street, Boston, MA 02131, United States Roslindale Roslindale NaN Boston MA 02131 Boston Boston, MA US United States 42.286241 -71.134374 t Apartment Private room 2 1.0 1.0 1.0 Real Bed {TV,Internet,"Wireless Internet","Air Conditio... NaN $65.00 $400.00 NaN $95.00 $10.00 0 $0.00 2 15 a week ago NaN 26 54 84 359 2016-09-06 36 2014-06-01 2016-08-13 94.0 10.0 9.0 10.0 10.0 9.0 9.0 f NaN NaN t moderate f f 1 1.30
2 6976 https://www.airbnb.com/rooms/6976 20160906204935 2016-09-07 Mexican Folk Art Haven in Boston Come stay with a friendly, middle-aged guy in ... Come stay with a friendly, middle-aged guy in ... Come stay with a friendly, middle-aged guy in ... none The LOCATION: Roslindale is a safe and diverse... I am in a scenic part of Boston with a couple ... PUBLIC TRANSPORTATION: From the house, quick p... I am living in the apartment during your stay,... ABOUT ME: I'm a laid-back, friendly, unmarried... I encourage you to use my kitchen, cooking and... https://a2.muscache.com/im/pictures/6ae8335d-9... https://a2.muscache.com/im/pictures/6ae8335d-9... https://a2.muscache.com/im/pictures/6ae8335d-9... https://a2.muscache.com/im/pictures/6ae8335d-9... 16701 https://www.airbnb.com/users/show/16701 Phil 2009-05-11 Boston, Massachusetts, United States I am a middle-aged, single male with a wide ra... within a few hours 100% 88% t https://a2.muscache.com/im/users/16701/profile... https://a2.muscache.com/im/users/16701/profile... Roslindale 1 1 ['email', 'phone', 'reviews', 'jumio'] t t Ardale St., Boston, MA 02131, United States Roslindale Roslindale NaN Boston MA 02131 Boston Boston, MA US United States 42.292438 -71.135765 t Apartment Private room 2 1.0 1.0 1.0 Real Bed {TV,"Cable TV","Wireless Internet","Air Condit... NaN $65.00 $395.00 $1,350.00 NaN NaN 1 $20.00 3 45 5 days ago NaN 19 46 61 319 2016-09-06 41 2009-07-19 2016-08-05 98.0 10.0 9.0 10.0 10.0 9.0 10.0 f NaN NaN f moderate t f 1 0.47
3 1436513 https://www.airbnb.com/rooms/1436513 20160906204935 2016-09-07 Spacious Sunny Bedroom Suite in Historic Home Come experience the comforts of home away from... Most places you find in Boston are small howev... Come experience the comforts of home away from... none Roslindale is a lovely little neighborhood loc... Please be mindful of the property as it is old... There are buses that stop right in front of th... The basement has a washer dryer and gym area. ... We do live in the house therefore might be som... - The bathroom and house are shared so please ... https://a2.muscache.com/im/pictures/39764190-1... https://a2.muscache.com/im/pictures/39764190-1... https://a2.muscache.com/im/pictures/39764190-1... https://a2.muscache.com/im/pictures/39764190-1... 6031442 https://www.airbnb.com/users/show/6031442 Meghna 2013-04-21 Boston, Massachusetts, United States My husband and I live on the property. He’s a... within a few hours 100% 50% f https://a2.muscache.com/im/pictures/5d430cde-7... https://a2.muscache.com/im/pictures/5d430cde-7... NaN 1 1 ['email', 'phone', 'reviews'] t f Boston, MA, United States NaN Roslindale NaN Boston MA NaN Boston Boston, MA US United States 42.281106 -71.121021 f House Private room 4 1.0 1.0 2.0 Real Bed {TV,Internet,"Wireless Internet","Air Conditio... NaN $75.00 NaN NaN $100.00 $50.00 2 $25.00 1 1125 a week ago NaN 6 16 26 98 2016-09-06 1 2016-08-28 2016-08-28 100.0 10.0 10.0 10.0 10.0 10.0 10.0 f NaN NaN f moderate f f 1 1.00
4 7651065 https://www.airbnb.com/rooms/7651065 20160906204935 2016-09-07 Come Home to Boston My comfy, clean and relaxing home is one block... Clean, attractive, private room, one block fro... My comfy, clean and relaxing home is one block... none I love the proximity to downtown, the neighbor... I have one roommate who lives on the lower lev... From Logan Airport and South Station you have... You will have access to the front and side por... I love my city and really enjoy sharing it wit... Please no smoking in the house, porch or on th... https://a1.muscache.com/im/pictures/97154760/8... https://a1.muscache.com/im/pictures/97154760/8... https://a1.muscache.com/im/pictures/97154760/8... https://a1.muscache.com/im/pictures/97154760/8... 15396970 https://www.airbnb.com/users/show/15396970 Linda 2014-05-11 Boston, Massachusetts, United States I work full time for a public school district.... within an hour 100% 100% t https://a0.muscache.com/im/users/15396970/prof... https://a0.muscache.com/im/users/15396970/prof... Roslindale 1 1 ['email', 'phone', 'reviews', 'kba'] t t Durnell Avenue, Boston, MA 02131, United States Roslindale Roslindale NaN Boston MA 02131 Boston Boston, MA US United States 42.284512 -71.136258 t House Private room 2 1.5 1.0 2.0 Real Bed {Internet,"Wireless Internet","Air Conditionin... NaN $79.00 NaN NaN NaN $15.00 1 $0.00 2 31 2 weeks ago NaN 13 34 59 334 2016-09-06 29 2015-08-18 2016-09-01 99.0 10.0 10.0 10.0 10.0 9.0 10.0 f NaN NaN f flexible f f 1 2.25
In [6]:
len(listings)
Out[6]:
3585

At the moment our listings have several geospatial variables, the most important of which are longitude and latitude, which give the exact coordinates of the BnB in question. This means that it's easy for us to, say, plot every BnB location on a map:

In [525]:
plt.scatter(listings['longitude'], listings['latitude'])
Out[525]:
<matplotlib.collections.PathCollection at 0x34240470>

Chances are you've already done this before, and it's a perfectly adequate way to get started working with locations. In this plot we see...not much, really. If you're very intimately familiar with the layout of the city of Boston, you will probably be able to make sense of some of these clusters which are, to me, not being from the city, totally mysterious.

In other words, this plot is missing something important: geospatial context.

Additionally, this display is unprojected—it's displayed in terms of raw coordinates. The amount of distance contained in a coordinate degree varies greatly depending on where you are, so this naive plot potentially pretty badly distorts distances.

We'll come back to the projection issue later; for now, there's an easy fix for both these problems.

Enter mplleaflet. mplleaflet is a tool that automatically takes a coordinate matplotlib plot of any kind and places it on top of a leaflet slippy map. The best part is that it's just one additional line of code. Just throw mplleaflet.display() after generating your plot to drop it inline in your Jupyter notebook:

In [526]:
import mplleaflet

sample = listings.sample(1000)
plt.scatter(sample['longitude'], sample['latitude'])

mplleaflet.display()
Out[526]: