Prototype to Production: Python Tools for Rapid Web Interface Development

Authors

Scott D Christensen and Marvin S Brown

Contact: [email protected]

COVID Tethys App Demo

Abstract

An interactive web application can take a powerful and complex Python workflow to the next level, drastically improve its utility, and enhance its ability to visualize results. The increased capabilities of open-source Python libraries have made the transition from Jupyter notebooks to production-ready web applications easier than ever. This presentation will demonstrate how to prototype a web application in the notebook environment, and then easily deploy it as a stand-alone web application with Panel. Next, we will show how Panel applications can be transitioned to a fully-featured web application in the Tethys Platform.

Introduction

Our goal is to be able to rapidly develop web interfaces for engineering and modeling workflows. Jupyter notebooks provide a good starting point to iterate on ideas and start to prototype visualizations. However, we ultimately want a customized, self-contained, interactive web app. Enhancements that have been made to the Panel library have made it easy to transition from prototyping in a Jupyter environment to a stand-alone Bokeh app. Additional enhancements to Bokeh and to Tethys Platform have made it possible to then transition your app into the Tethys environment.

App Spectrum

Demonstration

To demonstrate the use of these tools we will show a dashboard for exploring the COVID-19 data from The Johns Hopkins Center for Systems Science and Engineering.

We start with reading in the data and creating a visualization in a notebook:

In [ ]:
from pathlib import Path

import pandas as pd
import numpy as np
import xarray as xr
from pyproj import Transformer
import holoviews as hv
import param
import panel as pn

hv.extension('bokeh')

Read in Data

In [ ]:
VARS = ['Confirmed', 'Deaths', 'Recovered']

data_dir = Path('../COVID-19/csse_covid_19_data').resolve()

time_series_path = data_dir / 'csse_covid_19_time_series' / 'time_series_covid19_{VAR}_global.csv'

dfs = list()
dims = ['Aggregation', 'Quantity', 'Country', 'Date']
transformer = Transformer.from_crs("epsg:4326", "epsg:3857")
for var in VARS:
    df = pd.read_csv(time_series_path.as_posix().format(VAR=var.lower()))
    df.rename({'Country/Region': 'Country'}, axis=1, inplace=True)
    lat_lon = df[['Country', 'Lat', 'Long']]
    lat_lon = lat_lon.groupby('Country').first()
    lat_lon['x_y'] = lat_lon.apply(lambda row: transformer.transform(row.Lat, row.Long), axis=1)
    lat_lon['x'] = lat_lon.x_y.apply(lambda z: z[0])
    lat_lon['y'] = lat_lon.x_y.apply(lambda z: z[1])
    df.drop(['Lat', 'Long'], axis=1, inplace=True)
    df = df.groupby('Country').sum()
    df.insert(0, 'y', lat_lon['y'])
    df.insert(1, 'x', lat_lon['x'])
    df.sort_values(by='Country', inplace=True)
    dfs.append(df)
data_vars = [df.iloc[:, 2:].values for df in dfs]
data_vars.append(data_vars[0] - data_vars[1] - data_vars[2])  # active cases
VARS.append('Active')
data_vars = np.stack(data_vars)
data_vars = np.stack((data_vars, np.diff(data_vars, axis=-1, n=1, prepend=0)))  # daily changes
data_vars = (data_vars + np.absolute(data_vars)) / 2
data_vars = {'counts': (dims, data_vars)}

coords = dict(
    Country=('Country', df.index),
    Date=pd.to_datetime(df.columns.tolist()[2:]),
    x=('Country', df['x']),
    y=('Country', df['y']),
    Quantity=VARS,
    Aggregation=['Totals', 'Daily']
)
data = xr.Dataset(data_vars=data_vars, coords=coords)
data

Create Plot

In [ ]:
kdims = ['Date', 'Quantity', 'Aggregation']
grouped_data = data.sum('Country')
selected_data = hv.Dataset(
    grouped_data, 
    kdims=kdims, 
    vdims=['counts']).aggregate(dimensions=['Date', 'Quantity'], function=np.sum)
        
curves = selected_data.to(hv.Curve, 'Date', ).options(tools=['hover'], show_grid=True)
overlay = ['Quantity']

if overlay:
    curves = curves.overlay(overlay).options(legend_position='top_left', )
curves.options(
    responsive=True,
    height=800,
    xrotation=60,
)

Create Panel Dashboard

Using Panel we can convert our visualization into an interactive dashboard.

In [ ]:
class CovidPlotter(param.Parameterized):
    plot_type = param.ObjectSelector(default='Curve', objects=['Bar', 'Curve'], precedence=0.1)
    aggregation = param.ObjectSelector(default='Totals', objects=data.coords['Aggregation'].values, precedence=0.2)
    quantity = param.ObjectSelector(default='All', objects=['All'] + list(data.coords['Quantity'].values), precedence=0.2)
    groupby = param.ObjectSelector(default='Global', precedence=0.4, objects=['Global', 'Country'], label='Group By')
    countries = param.ListSelector(default=['China', 'US', 'Italy', 'France'], objects=np.unique(data.coords['Country'].values))
    # states = param.ListSelector(default=['Utah', 'Mississippi', 'California'], objects=[])
    selected_data = param.ClassSelector(hv.Dataset, precedence=-1)

    def __init__(self, **params):
        super().__init__(**params)
        self.dimensions = None
        # self.update_states()
        self.select_data()
        
    @param.depends('plot_type', watch=True)
    def update_defaults(self):
        if self.plot_type == 'Bar':
            self.aggregation = 'Daily'
            self.quantity = 'Confirmed'
            self.groupby = 'Global'
        else:
            self.aggregation = 'Totals'
            self.quantity = 'All'
    
    @param.depends('groupby', 'aggregation', 'quantity', 'countries', 'plot_type', watch=True)
    def select_data(self):
        kdims = ['Date', 'Quantity', 'Aggregation']
        self.dimensions = ['Date']
        select_kwargs = dict(Aggregation=self.aggregation)
        self.param.countries.precedence = -1
        # self.param.states.precedence = -1
        
        if self.groupby == 'Global':
            grouped_data = data.sum('Country')
        elif self.groupby == 'Country':
            grouped_data = data
            kdims.append('Country')
            self.param.countries.precedence = 1
            if len(self.countries) > 1 or self.plot_type == 'Curve':
                self.dimensions.append('Country')
            select_kwargs['Country'] = self.countries
        
        if self.quantity == 'All':
            self.dimensions.append('Quantity')
        else:
            select_kwargs['Quantity'] = self.quantity
            
        selected_data = hv.Dataset(grouped_data, kdims=kdims, vdims=['counts']).select(**select_kwargs).aggregate(dimensions=self.dimensions, function=np.sum)
        self.selected_data = selected_data
    
    @param.depends('selected_data')
    def curves(self):
        curves = self.selected_data.to(hv.Curve, 'Date', ).options(tools=['hover'], show_grid=True)
        overlay = list()
        if self.quantity == 'All':
            overlay.append('Quantity')
        if self.groupby == 'Country':
            overlay.append('Country')
        if self.groupby == 'State':
            overlay.append('Province_State')
            
        if overlay:
            curves = curves.overlay(overlay).options(legend_position='top_left', )
        return curves.options(
                responsive=True,
                height=800,
                xrotation=60,
            )
    
    @param.depends('selected_data')
    def bars(self):
        kdims = self.dimensions
        return hv.Bars(self.selected_data, kdims=kdims).opts(
            responsive=True,
            height=800,
            stacked=False, 
            show_legend=False, 
            xrotation=60,
            tools=['hover'],
        )
    
    @param.depends('plot_type')
    def plot(self):
        return dict(
            Bar=self.bars,
            Curve=self.curves,
        )[self.plot_type]
    
    def panel(self):
        return pn.Row(
            pn.Param(self,
                     widgets={
                         'countries': {'height': 550},
                     },
                     show_name=False),
            self.plot,
        )

CovidPlotter().panel()

Continuing this process we created several visualizations that are all tied into a single Panel app. We exported the code into external Python files that can then be simply imported and executed in a Notebook:

In [ ]:
from covid_dashboard import CovidDashboard

Complete Panel App

In [ ]:
CovidDashboard().panel()

Converting to a Stand-alone Bokeh App

When running a notebook locally you can launch a stand-alone Bokeh app right from the notebook by adding a .show() to any Panel output.

In our case this would be:

CovidDashboard().panel().show()

Alternatively you can create a Python script, main.py, similar to the following:

# filename: main.py

from covid_dashboard import CovidDashboard

cd = CovidDashboard().panel()
cd.servable()

If the main.py script were located in a directory called bokeh_app/ then this Bokeh application could be launched from the commandline like this:

panel serve /path/to/bokeh_app/

To see this served as a Bokeh app in Binder see the following link:

Binder

Converting to a Production Tethys App

Converting the app to Tethys is not quite as simple as moving from the notebook to a Bokeh app, but it is still very easy and requires minimal code.

After creating a Tethys App Scaffold, then a handlers.py file needs to be added that is similar to the main.py file mentioned above. It should contain code that is something like this:

from bokeh.document import Document

from covid_dashboard import CovidDashboard

def handler(doc: Document) -> None:
    cd = CovidDashboard().panel()
    cd.server_doc(doc)

Next the default home controller in the controllers.py file should be modified to contain the following:

def home(request):
    """
    Controller for the app home page.
    """
    script = server_document(request.get_full_path())

    context = {
        'script': script,
    }

    return render(request, 'covid/home.html', context)

Finally, the handler needs to be registered with the default home UrlMap in app.py:

UrlMap(
        name='home',
        url='covid',
        controller='covid.controllers.home',
        handler='covid.handlers.handler',
        handler_type='bokeh',
    )

With those changes the app will then be integrated into a Tethys app. Additional changes can then be made to take advantage of the Tethys framework.

COVID Tethys App Demo

In [ ]: