Explore a Global News Media Dataset using Jupyter Widgets

Objective: Use the Descartes Labs Vectors Service to query the Global Database of Events, Language and Tone (GDELT Project)

This tutorial demonstrates loading data through the Vectors Service and plotting it in an embedded map powered by Jupyter Widgets and ipyleaflet. To learn more about GDELT, visit: https://www.gdeltproject.org/ It is worth noting the GDELT dataset loaded for this example is only a single snap-shot in time and is not regularly updated. To learn more about ipyleaflet, visit: https://ipyleaflet.readthedocs.io/en/latest/api_reference/widget_control.html

In [1]:
from ipyleaflet import Map, basemaps, basemap_to_tiles, TileLayer, DrawControl, \
    SplitMapControl, FullScreenControl, GeoJSON, WidgetControl
from ipywidgets import HTML
import ipywidgets as w
from ipyleaflet import DrawControl, SplitMapControl, FullScreenControl, LayersControl, Marker, MarkerCluster
from descarteslabs import Raster
import descarteslabs as dl
import time

Here, we prepare two tiled layers for the map background using the Descartes Labs tiles endpoint. One true color composite and one NDVI composite.

In [2]:
tiles = Raster("https://platform.descarteslabs.com/tiles/v2/")

before_options = {"products":["sentinel-2:L1C"],
           "bands":["red","green","blue","alpha"],
           "scales":[[0,4000],[0,4000],[0,4000],[0,1]],
           "start_time":"2019-03-18T00:00:00Z",
           "end_time":"2019-03-29T00:00:00Z",
           "date":"acquired",
           "processing_level":"toa"}

after_options = {"products":["sentinel-2:L1C"],
           "bands":["ndvi","alpha"],
           "scales":[[32768,65535],[0,1]],
           "start_time":"2019-03-18T00:00:00Z",
           "end_time":"2019-03-29T00:00:00Z",
           "date":"acquired",
           "processing_level":"toa"}

before_tile = TileLayer()
before_tile.url = tiles.session.post("xyz", json=before_options).json()['url']
before_tile.name = "RGB"

after_tile = TileLayer()
after_tile.url = tiles.session.post("xyz", json=after_options).json()['url']
after_tile.name = "NDVI"

Next, we create a function that allows you to query the GDELT dataset based on the ipyleaflet map's viewport. The function renders the returned data points on the embedded map that display articles once run.

In [3]:
layers = {}
current_features = []
clicked = []
debug_view = w.Output(layout={'border': '1px solid black'})
current_features = []

@debug_view.capture(clear_output=True)
def load_gdelt(*args, **kwargs):
    clicked.append((args, kwargs))
    print("clicked")
    clicked.append(m)

    fc = dl.vectors.FeatureCollection("1be52be97ed54c22bfc3a09f4b64155")
    fc.filter(geometry=m.bounds_polygon)
    bound_geom = [list(p)[::-1] for p in m.bounds_polygon[::-1]]
    bound_geom.append(bound_geom[0])    
    geom_json = {'type':'Polygon', 'coordinates':[bound_geom]}
    print("about to query")
    
    markers = []
    t1 = -time.time()
    for FF in fc.filter(geometry=geom_json).limit(100).features():
        f = FF.geojson
        marker = Marker(location=f['geometry']['coordinates'][::-1])
        popup = HTML()
        url = f['properties']['SOURCEURL']
        popup.value = """<a href={} target="_blank">{}</a>""".format(url, url)
        marker.popup = popup
        markers.append(marker)
        current_features.append(f) 
    t1 += time.time()
    print("gathered features in {}".format(t1))

    marker_cluster = MarkerCluster(markers=markers)
    
    m.add_layer(marker_cluster)
    layers['gdelt'] = marker_cluster
    clicked.append("4")
    
button = w.Button(
    description='Search GDELT',
    disabled=False,
    button_style='', # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Search',
)
button.on_click(load_gdelt)
wc = WidgetControl(widget = button, position='bottomright')

debug_view

The next code block renders the ipyleaflet map with the tiled layers we defined above. You can toggle between the true color composite, NDVI composite, and a base map using the layer drop-down in the top right hand corner. You can expand the view to the full extent of the screen using the expand button in the bottom right hand corner.

After running the code block, set the view port to represent your area of interest. Scroll to the bottom of the embedded map and click "Search GDELT".

In [4]:
m = Map(center=(35.6824, -105.581), zoom=9, scroll_wheel_zoom=True, min_zoom=4)
m.add_layer(before_tile)
m.add_layer(after_tile)

m.layout.height = '600px'

draw_control = DrawControl()
m.add_control(draw_control)
m.add_control(FullScreenControl())
m.add_control(LayersControl())

draw_events = []

@debug_view.capture(clear_output=True)
def callback(*args, **kwargs):
    draw_events.append(kwargs['geo_json'])
    
    
draw_control.on_draw(callback)

m.add_control(wc)
m
In [5]:
draw_events
Out[5]:
[]