Choropleth maps are maps in which well-defined regions of a map are coloured according to a particular indicator value.
To generate such a map, we need three things:
In terms of getting boundary lines, the geojson data format (.json
, .geojson
) is increasingly used as a lightweight standard for trasnporting geodata in web apps. Other formats include KML (.kml
) and ESRI shapefiles (.shp
).
One useful source of boundary line data for UK administrative boundaries is from the MySociety MapIt website/service.
For example, we can identify a range of adminsitrative boundaries for the Isle of Wight by looking up the Isle of Wight Council postcode on MapIt: PO30 1UD.
From there we can get a link to an adminstrative region, such as the Isle of Wight parliamentary constituency. You can also use MapIt to find other regions, such as adjoining regions, or regions covered by or covering a particular area.
We can then get the geometry file as geojson and render it using folium.
Martin Chorley has collected together on Github a range of useful shapefiles in geojson and TopoJSON formats that describe various electoral boundaries martinjc/UK-GeoJSON.
#http://bit.ly/ccweek7code
from IPython.display import HTML
import folium
def inline_map(map):
"""
Embeds the HTML source of the map directly into the IPython notebook.
This method will not work if the map depends on any files (json data). Also this uses
the HTML5 srcdoc attribute, which may not be supported in all browsers.
"""
map._build_map()
return HTML('<iframe srcdoc="{srcdoc}" style="width: 100%; height: 510px; border: none"></iframe>'.format(srcdoc=map.HTML.replace('"', '"')))
def embed_map(map, path="map.html"):
"""
Embeds a linked iframe to the map into the IPython notebook.
Note: this method will not capture the source of the map into the notebook.
This method should work for all maps (as long as they use relative urls).
"""
map.create_map(path=path)
return HTML('<iframe src="files/{path}" style="width: 100%; height: 510px; border: none"></iframe>'.format(path=path))
geojson_url_iw='http://mapit.mysociety.org/area/65791.geojson'
iw_map = folium.Map(location=[50.666, -1.37], zoom_start=11)
iw_map.geo_json(geo_path= geojson_url_iw)
embed_map(iw_map,'iw_map.html')
See if you can map some other areas of different administrative type.
As we did last week, we can patch a standalone HTML file containing the map created/saved using embed_map()
with the following routine:
def patcher(fn='map.html'):
f=open(fn,'r')
html=f.read()
f.close()
html=html.replace('"//','"http://')
f=open(fn,'w')
f.write(html)
f.close()
patcher('iw_map.html')
You can play with the fill colour and transparency level of the data using the fill_color
and fill_opacity
parameters to the .geo_json()
function.
#We can also move up a level to the South East region (http://mapit.mysociety.org/area/11811.html) for example
geojson_url_se='http://mapit.mysociety.org/area/11811.geojson'
se_map = folium.Map(location=[51.4, -1], zoom_start=8)
se_map.geo_json(geo_path= geojson_url_se,fill_color='green',fill_opacity=0.3)
embed_map(se_map,'se_map.html')
Experiment with various fill colour and transparency settings.
How would you add additional regions to the map?
#See if you can figure out how to add several regions to the map, perhaps with different colours...
#We can add additional layers to the map object, just as we added additional markers to the map last week
#So for example, add an IW layer to the SE map...
se_map.geo_json(geo_path= geojson_url_iw,fill_color='orange',fill_opacity=0.3)
embed_map(se_map,'se_map.html')
If we look at the page for Newport - http://mapit.mysociety.org/area/61005.html - the "county town" of the Isle of Wight, we see that we can find a list of additional geographies covered by that region:http://mapit.mysociety.org/area/61005/covers.html.
We can futher filter these to geographies of a particular type, for example: http://mapit.mysociety.org/area/61005/covers.html?type=UTE
We can also get this data back as JSON: http://mapit.mysociety.org/area/61005/covers?type=UTE
#Use the requests library to load the json data
import requests
import json
regions='http://mapit.mysociety.org/area/61005/covers?type=UTE'
jsondata=json.loads( requests.get(regions).content )
jsondata
#We can iterate through the covered regions to grab their ids
for key in jsondata:
print(key)
Can you think of a way of taking the Isle of Wight map (iw_map
above) and adding to it white filled boundaries for the URE regions covered by Newport?
#Using iw_map, try to add white colour filled shapes for each UTE region in Newport
for key in jsondata:
tmp_url='http://mapit.mysociety.org/area/{}.geojson'.format(key)
iw_map.geo_json(geo_path= tmp_url,fill_color='white',fill_opacity=1)
embed_map(iw_map,'iw_map.html')
Something topical...
Chris Hanretty et al. are making election forecasts based on aggregated poll data available at electionforecast.co.uk.
The forecast data is published as an HTML table at http://www.electionforecast.co.uk/tables/predicted_probability_by_seat.html.
HOW WOULD YOU LOAD THIS DATA INTO A PANDAS DATAFRAME? (HINT: do you remember how to use .read_html()?)
#See if you can grab the data from http://www.electionforecast.co.uk/tables/predicted_probability_by_seat.html
#into a pandas dataframe...
#We can grab the data into a data frame by scraping the tabular HTML data from the URL directly
#The pandas .read_html() function can accept a URL and will return a list of tables scraped from the page
import pandas as pd
df=pd.read_html('http://www.electionforecast.co.uk/tables/predicted_probability_by_seat.html')
#We index into the table list response to get the table we want...
df[0][:3]
If you look at the election forecast data you wil see the official name of the constituency listed in the Seat column.
Now let's grab some shapefiles corresponding to the Westminster consituencies.
#Grab Westminster parliamentary constituency shapefiles from Martin Chorley's github repository
import requests
url='https://github.com/martinjc/UK-GeoJSON/blob/master/json/electoral/gb/wpc.json?raw=true'
r = requests.get(url)
#And save it to the local directory as the file wpc.json
with open("wpc.json", "wb") as code:
code.write(r.content)
#Free up memory...
r=None
As well as setting geo_path=URL
, we can also set geo_path=LOCAL_FILE
.
See if you can plot a map showing the Westminster Parliamentary Constituency boundaries
Look at the documentation for more details. For example:
?iw_map.geo_json
#Now try plotting a map of UK Westminster Parliamentary Constituency boundaries
#Here's one way...
#Create a base map
wpc_map = folium.Map(location=[55, 0], zoom_start=5)
wpc_map.geo_json(geo_path= "wpc.json",fill_color='orange',fill_opacity=0.3)
embed_map(wpc_map,'wpc_map.html')
If you look at the structure of the wpc.json file, you will see that it takes the following form:
That is, it contains a list of features
each of which have a set of properties
that includes one called PCON13NM
that contains the parlimentary constituency name.
wpc_map.geo_json(geo_path='wpc.json', data=df[0],data_out='data_lab.json', columns=['Seat', 'Labour'],
key_on='feature.properties.PCON13NM',threshold_scale=[0, 20, 40, 60, 80, 100],
fill_color='OrRd')
embed_map(wpc_map)
How would you create a map to show the likelihood of the Conservatives winning each seat?
#Create a forecast map showing the likelihood of the Conservatives taking each seat
#Use a different colour theme such as GnBu for the colour scale
#Other colour schemes can be found from the library docmentation - http://folium.readthedocs.org/en/latest/
The folium library does not allow us to pass in discrete coloursor use a categorical mapping when binding data. Instead, if we wanted to plot a map that showed the colour of the most likely winner of a seat, we would need to:
forecast_m =pd.melt(df[0], id_vars=['Seat','Region'],
value_vars=['Conservatives','Labour','Liberal Democrats','SNP','Plaid Cymru','Greens','UKIP','Other'],
var_name='Party', value_name='forecast')
forecast_m[:3]
likelyparty=forecast_m.sort('forecast', ascending=False).groupby('Seat', as_index=False).first()
likelyparty[:3]
partycolours={'Conservatives':'blue',
'Labour':'red',
'Liberal Democrats':'yellow',
'SNP':'orange',
'Plaid Cymru':'pink',
'Greens':'green',
'UKIP':'purple',
'Other':'black'}
likelyparty['colour']=likelyparty['Party'].map(partycolours)
likelyparty[:3]
The next thing we need to do is to be able to get hold of the shape information for each constituency. folium handled this for us automatically when we used the .geo_json()
function, but this time we need to extract a separate boundary for each constituency.
Note also that we can pass a geojson string into folium using geo_str=GOEJSON_STRING
rather than passing in a filename or URL using geo_path
.
import json
jj=json.load(open('wpc.json'))
for c in jj['features'][:3]:
print(c['properties']['PCON13NM'])
likelyparty[likelyparty['Seat']=='Aberavon']
#If we convert the dataframe to a dict, we can lookup colour by seat
#Set the index to be the seat, then we get a nested dict keyed at the top level by column and then by seat
likelyparty_dict=likelyparty.set_index('Seat').to_dict()
likelyparty_dict['colour']
#Can you remember how we added boundaries for areas in Newport to the Isle of Wight map...
#The following approach uses a similar principle
forecast_map = folium.Map(location=[55, 0], zoom_start=5)
#Iterate through each constituency in the geojson file getting each feature (constituency shapefile) in turn
for c in jj['features']:
#The geojson format requires that features are provided in a list and a FeatureCollection defined as the type
#So we wrap the feature definition for each constituency in the necessary format
geodata= {"type": "FeatureCollection", "features": [c]}
#Get the name of the seat for the current constituency
seat=c['properties']['PCON13NM']
#We can now lookup the colour
colour= likelyparty_dict['colour'][seat]
forecast_map.geo_json(geo_str= json.dumps(geodata),fill_color=colour,fill_opacity=1)
embed_map(forecast_map,'forecast_map.html')
folium makes working with maps using geojson relatively straightforward. However, if your shapefiles come in the form of ESRI .shp
format files, folium cannot work with them directly.
A workaround is to convert the .shp
file to a geojson file for use directly in folium.
The Python shapefile
library (available as pyshp) helps us do what we need.
(Note that there are more powerful tools available for working with geo-data, including a pandas extension called geopandas, but many of them require additional non-python packages libraries installing on your computer before they can be used.)
#You will probably ned to install pyshp which contains the shapefile package
#!pip install pyshp
import shapefile
#I got a few cribs for how to use this package to generate JOSN file from here
#https://github.com/mlaloux/PyShp-as-Fiona--with-geo_interface-
def records(filename):
# generator
reader = shapefile.Reader(filename)
fields = reader.fields[1:]
field_names = [field[0] for field in fields]
for sr in reader.shapeRecords():
geom = sr.shape.__geo_interface__
atr = dict(zip(field_names, sr.record))
yield dict(geometry=geom,properties=atr)
#shapefiles for African countries can be found here:
#http://www.mapmakerdata.co.uk/library/stacks/Africa/
#eg I downloaded a file for Botswana via http://www.mapmakerdata.co.uk/library/stacks/Africa/Botswana/index.htm
#from http://www.mapmakerdata.co.uk/library/stacks/Africa/Botswana/BOT_admin_SHP.zip
#and then unzipped the file
f='/Users/ajh59/Downloads/BOT_admin_SHP/BOT.shp'
#Use the records() function to parse the shapefile
c= records(f)
#Here's what a single record looks like
c.next()
import json
#The following routine will generate a json file equivalent of the shapefile
def geojsonify(shpfile,fn='gtest.json'):
geojson = open(fn, "w")
features=[row for row in records(shpfile)]
for row in features:
row["type"]="Feature"
geojson.write(json.dumps(dict(type='FeatureCollection',features=features)))
geojson.close()
!ls
geojsonify(f,'gtestX.json')
map_shp = folium.Map(location=[-25, 15], zoom_start=5)
map_shp.geo_json(geo_path='gtestX.json')
inline_map(map_shp)
Lint tools are tools that check whether or not something is correctly formatted. If geojson is not correctly formatted it won't rended on a map. geojsonlint is one tool for checking that your geojson is well formed.
If you want to create your own, jhand drawn, shapefile, there are a couple of tools that can help you do it... For example, geojson.io or a new one from Google, Simple GeoJSON Editor.