Intake catalog example

We would like to more easily interact with the datasets available at data.lacity.org and geohub.lacity.org.

These data portals have catalogs available at the data.json endpoints, which are specified in the DCAT (https://project-open-data.cio.gov/v1.1/schema) catalog format.

We have been working on an intake catalog source that adapts a DCAT catalog to an intake catalog, along with specifying how to load datasets.

Begin by importing intake:

In [1]:
%matplotlib inline
import intake

The DCAT catalogs for the City of Los Angeles open data portals are specified in catalog.yml. We can optionally give an item argument, which will filter the catalog to only include the selected items.

We use intake to load the catalog into memory:

In [2]:
catalog = intake.open_catalog('catalogs/demo.yml')
print(list(catalog))
['la_open_data', 'la_geohub']

Let's load some data relating to Los Angeles' bike infrastructure. The GeoHub dataset has already had some entries selected:

In [3]:
geohub = catalog.la_geohub
open_data = catalog.la_open_data
for key, entry in geohub.items():
    display(entry)

City of Los Angeles Bikeways

ID: http://geohub.lacity.org/datasets/2602345a7a8549518e8e3c873368c1d9_0

Issued: 2018-05-15T14:24:19.000Z

Last Modified: 2019-07-17T16:17:11.000Z

Publisher: unknown

License: http://geohub.lacity.org/datasets/2602345a7a8549518e8e3c873368c1d9_0/license.json

Download URL: http://geohub.lacity.org/datasets/2602345a7a8549518e8e3c873368c1d9_0.geojson

City of Los Angeles network of on-street and off-street bikeways. Included Bike Path Class 1, Bike Lane Class 2, Bike Route/ Sharrow Class 3, Protected Bike Lanes Class 4.

City Boundary

ID: http://geohub.lacity.org/datasets/09f503229d37414a8e67a7b6ceb9ec43_7

Issued: 2015-11-14T01:23:22.000Z

Last Modified: 2018-04-06T16:48:27.000Z

Publisher: unknown

License: http://geohub.lacity.org/datasets/09f503229d37414a8e67a7b6ceb9ec43_7/license.json

Download URL: http://geohub.lacity.org/datasets/09f503229d37414a8e67a7b6ceb9ec43_7.geojson

City Boundary 

Bike Racks

ID: http://geohub.lacity.org/datasets/3b022cced9704108af157d3d5eedb268_2

Issued: 2018-11-26T21:31:23.000Z

Last Modified: 2018-11-26T21:31:41.000Z

Publisher: unknown

License: http://geohub.lacity.org/datasets/3b022cced9704108af157d3d5eedb268_2/license.json

Download URL: http://geohub.lacity.org/datasets/3b022cced9704108af157d3d5eedb268_2.geojson

Bike Corrals on-street bicycle parking facility that can accommodate many more bicycles than a typical sidewalk rack. Bike Racks installed where possible in the public right-of-way bike parking as close to building entrances as possible to increase security and make travelling by bike convenient. Bike Repair stations provide basic bicycle repair capability to business districts and corridors that cater to bicyclists. Repair Stations feature a stand to mount a bicycle and contain the basic tools needed to perform do-it-yourself bicycle repair including screwdrivers, wrenches, and hex tools. Repair stations also feature a heavy duty bicycle pump with a pump head for both schrader and presta valves and connect users to detailed instructions for a wide variety of bicycle repairs-just a smart phone scan away.

The open data catalog, on the other hand, is pretty long:

In [4]:
len(list(open_data))
Out[4]:
285

We can make this shorter by filtering it for the "boundary" keyword:

In [5]:
open_data_boundary = open_data.search('boundary')
len(list(open_data_boundary))
Out[5]:
4
In [6]:
for entry_id, entry in open_data_boundary.items():
    display(entry)

Posted Street Sweeping Routes

ID: https://data.lacity.org/api/views/krk7-ayq2

Issued: 2014-05-28

Last Modified: 2016-11-30

Publisher: data.lacity.org

License: http://creativecommons.org/publicdomain/zero/1.0/legalcode

Download URL: https://data.lacity.org/api/views/krk7-ayq2/rows.csv?accessType=DOWNLOAD

Motor sweeping program: route #s, boundaries and times for posted routes only. Maps are available as downloadable PDFs under the Attachments section of the About tab.

Area Planning Commission

ID: https://data.lacity.org/api/views/paqi-j4yv

Issued: 2014-05-23

Last Modified: 2019-08-01

Publisher: data.lacity.org

License: http://creativecommons.org/publicdomain/zero/1.0/legalcode

Download URL: https://data.lacity.org/api/geospatial/paqi-j4yv?method=export&format=Shapefile

Area Planning Commission Boundary

City Boundary of Los Angeles

ID: https://data.lacity.org/api/views/ppge-zfr4

Issued: 2016-05-26

Last Modified: 2017-10-27

Publisher: data.lacity.org

License: http://creativecommons.org/publicdomain/zero/1.0/legalcode

Download URL: https://data.lacity.org/api/geospatial/ppge-zfr4?method=export&format=Shapefile

Official City Boundary for the City of Los Angeles created and maintained by the Bureau of Engineering / GIS Mapping Division.

Port of Los Angeles Point of Interest

ID: https://data.lacity.org/api/views/vu99-34t2

Issued: 2014-05-27

Last Modified: 2016-11-29

Publisher: data.lacity.org

License: http://creativecommons.org/publicdomain/zero/1.0/legalcode

Download URL: https://data.lacity.org/api/geospatial/vu99-34t2?method=export&format=Shapefile

Location of lighthouses, museums, public arts, parks, restrooms and special interest within Port of Los Angeles' administrative boundary. Data will be refresh when there's a revision to the current points of interest.

Much more managable.

We can new read in some datasets into memory:

In [7]:
# The subsetted geohub catalog was able to rename the selected datasets
# so that they are nicer to read.
bike_racks = geohub.bike_racks.read()
bikeways = geohub.bikeways.read()
# The auto-generated open data catalog used the default identifier,
# which is a bit less nice to read.
city_boundary = open_data_boundary['https://data.lacity.org/api/views/ppge-zfr4'].read()

Let's inspect these new dataframes:

In [8]:
bikeways.head()
Out[8]:
OBJECTID_12 OBJECTID_1 OBJECTID_2 OBJECTID SECT_ID STREET_DES OTHER FROM_ TO_ ST_TYPE ... Project RetireType InputDate CHECKED Shape_Leng flag Shape_Le_1 Upgraded_F Shape__Length geometry
0 1001 967 891 12625 6500700 Collector Street WISCONSIN ST WISCONSIN PL/WISCONSIN ST Select ... None 0 77.680483 0 77.680483 77.680572 LINESTRING Z (-118.293080987425 34.01549153583...
1 1002 968 892 13419 6503700 Collector Street DEGNAN BL EDGEHILL DR Select ... None 0 319.462025 0 319.462025 319.462157 LINESTRING Z (-118.330100888092 34.01460809311...
2 1003 969 893 15965 6501700 Collector Street HARVARD BL HOBART BL Select ... None 0 320.005331 0 320.005331 320.005344 LINESTRING Z (-118.306738455446 34.01462223768...
3 1004 970 894 15728 6715900 Secondary Highway ANZAC AV WILMINGTON AV Select ... None 0 344.910378 0 344.910378 344.910411 LINESTRING Z (-118.238559018259 33.95327449103...
4 1005 971 895 14792 6332100 Major Highway - Class II AVERILL AV HARBOR VIEW AV Select ... None 0 292.805289 0 292.805289 292.805375 LINESTRING Z (-118.30776424806 33.736108417775...

5 rows × 37 columns

In [9]:
bike_racks.head()
Out[9]:
OBJECTID Status Side ARC_Zone LocID LocName Intersect_ Address_X City State ... RackColor Status_1 Notes Install_Da Priority Request Batch Addrees Zip_Code geometry
0 7001 M L 91364 381 Acapulco Canoga Ave 21326 Ventura Boulevard Woodland Hills CA ... Black Installed 21326 VENTURA BLVD 91364 POINT (-118.5964108110045 34.1674938768729)
1 7002 M L 91364 379 Liquor Store Alhama Dr. 21052 Ventura Boulevard Woodland Hills CA ... Black Installed 21052 VENTURA BLVD 91364 POINT (-118.5918492393963 34.16661162296765)
2 7003 M L 91364 378 Paolis Paralta Avenue 21020 Ventura Boulevard Woodland Hills CA ... Black Installed 21020 VENTURA BLVD 91364 POINT (-118.5916275254887 34.16658034149554)
3 7004 M R 91364 377 Ivy Glen Paralta Avenue 20929 Ventura Boulevard Woodland Hills CA ... Black Installed 20929 VENTURA BLVD 91364 POINT (-118.5892949355671 34.16669153771793)
4 7005 M L 91367 390 Subway Variel Ave 21032 Victory Boulevard Woodland Hills CA ... Black Installed 21032 VICTORY BLVD 91367 POINT (-118.5916133948874 34.18818774224465)

5 rows × 26 columns

These can all be plotted to show a view of Los Angeles bike infrastructure:

In [10]:
import matplotlib.pyplot as plt
fig, ax = plt.subplots(1, figsize=(16,16))
city_boundary.plot(ax=ax, color='darkgreen', alpha=0.2, linewidth=1, edgecolor='black')
bikeways.plot(ax=ax, color='navy', alpha=0.5, linewidth=1)
bike_racks.plot(ax=ax, color='maroon', alpha=0.5, markersize=1)
Out[10]:
<matplotlib.axes._subplots.AxesSubplot at 0x1eecee5d0f0>