Prototype to Production: Python Tools for Rapid Web Interface Development¶
Authors¶
Scott D Christensen and Marvin S Brown
Contact: [email protected]
Abstract¶
An interactive web application can take a powerful and complex Python workflow to the next level, drastically improve its utility, and enhance its ability to visualize results. The increased capabilities of open-source Python libraries have made the transition from Jupyter notebooks to production-ready web applications easier than ever. This presentation will demonstrate how to prototype a web application in the notebook environment, and then easily deploy it as a stand-alone web application with Panel. Next, we will show how Panel applications can be transitioned to a fully-featured web application in the Tethys Platform.
Introduction¶
Our goal is to be able to rapidly develop web interfaces for engineering and modeling workflows. Jupyter notebooks provide a good starting point to iterate on ideas and start to prototype visualizations. However, we ultimately want a customized, self-contained, interactive web app. Enhancements that have been made to the Panel library have made it easy to transition from prototyping in a Jupyter environment to a stand-alone Bokeh app. Additional enhancements to Bokeh and to Tethys Platform have made it possible to then transition your app into the Tethys environment.
Demonstration¶
To demonstrate the use of these tools we will show a dashboard for exploring the COVID-19 data from The Johns Hopkins Center for Systems Science and Engineering.
We start with reading in the data and creating a visualization in a notebook:
from pathlib import Path
import pandas as pd
import numpy as np
import xarray as xr
from pyproj import Transformer
import holoviews as hv
import param
import panel as pn
hv.extension('bokeh')
Read in Data¶
VARS = ['Confirmed', 'Deaths', 'Recovered']
data_dir = Path('../COVID-19/csse_covid_19_data').resolve()
time_series_path = data_dir / 'csse_covid_19_time_series' / 'time_series_covid19_{VAR}_global.csv'
dfs = list()
dims = ['Aggregation', 'Quantity', 'Country', 'Date']
transformer = Transformer.from_crs("epsg:4326", "epsg:3857")
for var in VARS:
df = pd.read_csv(time_series_path.as_posix().format(VAR=var.lower()))
df.rename({'Country/Region': 'Country'}, axis=1, inplace=True)
lat_lon = df[['Country', 'Lat', 'Long']]
lat_lon = lat_lon.groupby('Country').first()
lat_lon['x_y'] = lat_lon.apply(lambda row: transformer.transform(row.Lat, row.Long), axis=1)
lat_lon['x'] = lat_lon.x_y.apply(lambda z: z[0])
lat_lon['y'] = lat_lon.x_y.apply(lambda z: z[1])
df.drop(['Lat', 'Long'], axis=1, inplace=True)
df = df.groupby('Country').sum()
df.insert(0, 'y', lat_lon['y'])
df.insert(1, 'x', lat_lon['x'])
df.sort_values(by='Country', inplace=True)
dfs.append(df)
data_vars = [df.iloc[:, 2:].values for df in dfs]
data_vars.append(data_vars[0] - data_vars[1] - data_vars[2]) # active cases
VARS.append('Active')
data_vars = np.stack(data_vars)
data_vars = np.stack((data_vars, np.diff(data_vars, axis=-1, n=1, prepend=0))) # daily changes
data_vars = (data_vars + np.absolute(data_vars)) / 2
data_vars = {'counts': (dims, data_vars)}
coords = dict(
Country=('Country', df.index),
Date=pd.to_datetime(df.columns.tolist()[2:]),
x=('Country', df['x']),
y=('Country', df['y']),
Quantity=VARS,
Aggregation=['Totals', 'Daily']
)
data = xr.Dataset(data_vars=data_vars, coords=coords)
data
Create Plot¶
kdims = ['Date', 'Quantity', 'Aggregation']
grouped_data = data.sum('Country')
selected_data = hv.Dataset(
grouped_data,
kdims=kdims,
vdims=['counts']).aggregate(dimensions=['Date', 'Quantity'], function=np.sum)
curves = selected_data.to(hv.Curve, 'Date', ).options(tools=['hover'], show_grid=True)
overlay = ['Quantity']
if overlay:
curves = curves.overlay(overlay).options(legend_position='top_left', )
curves.options(
responsive=True,
height=800,
xrotation=60,
)
Create Panel Dashboard¶
Using Panel we can convert our visualization into an interactive dashboard.
class CovidPlotter(param.Parameterized):
plot_type = param.ObjectSelector(default='Curve', objects=['Bar', 'Curve'], precedence=0.1)
aggregation = param.ObjectSelector(default='Totals', objects=data.coords['Aggregation'].values, precedence=0.2)
quantity = param.ObjectSelector(default='All', objects=['All'] + list(data.coords['Quantity'].values), precedence=0.2)
groupby = param.ObjectSelector(default='Global', precedence=0.4, objects=['Global', 'Country'], label='Group By')
countries = param.ListSelector(default=['China', 'US', 'Italy', 'France'], objects=np.unique(data.coords['Country'].values))
# states = param.ListSelector(default=['Utah', 'Mississippi', 'California'], objects=[])
selected_data = param.ClassSelector(hv.Dataset, precedence=-1)
def __init__(self, **params):
super().__init__(**params)
self.dimensions = None
# self.update_states()
self.select_data()
@param.depends('plot_type', watch=True)
def update_defaults(self):
if self.plot_type == 'Bar':
self.aggregation = 'Daily'
self.quantity = 'Confirmed'
self.groupby = 'Global'
else:
self.aggregation = 'Totals'
self.quantity = 'All'
@param.depends('groupby', 'aggregation', 'quantity', 'countries', 'plot_type', watch=True)
def select_data(self):
kdims = ['Date', 'Quantity', 'Aggregation']
self.dimensions = ['Date']
select_kwargs = dict(Aggregation=self.aggregation)
self.param.countries.precedence = -1
# self.param.states.precedence = -1
if self.groupby == 'Global':
grouped_data = data.sum('Country')
elif self.groupby == 'Country':
grouped_data = data
kdims.append('Country')
self.param.countries.precedence = 1
if len(self.countries) > 1 or self.plot_type == 'Curve':
self.dimensions.append('Country')
select_kwargs['Country'] = self.countries
if self.quantity == 'All':
self.dimensions.append('Quantity')
else:
select_kwargs['Quantity'] = self.quantity
selected_data = hv.Dataset(grouped_data, kdims=kdims, vdims=['counts']).select(**select_kwargs).aggregate(dimensions=self.dimensions, function=np.sum)
self.selected_data = selected_data
@param.depends('selected_data')
def curves(self):
curves = self.selected_data.to(hv.Curve, 'Date', ).options(tools=['hover'], show_grid=True)
overlay = list()
if self.quantity == 'All':
overlay.append('Quantity')
if self.groupby == 'Country':
overlay.append('Country')
if self.groupby == 'State':
overlay.append('Province_State')
if overlay:
curves = curves.overlay(overlay).options(legend_position='top_left', )
return curves.options(
responsive=True,
height=800,
xrotation=60,
)
@param.depends('selected_data')
def bars(self):
kdims = self.dimensions
return hv.Bars(self.selected_data, kdims=kdims).opts(
responsive=True,
height=800,
stacked=False,
show_legend=False,
xrotation=60,
tools=['hover'],
)
@param.depends('plot_type')
def plot(self):
return dict(
Bar=self.bars,
Curve=self.curves,
)[self.plot_type]
def panel(self):
return pn.Row(
pn.Param(self,
widgets={
'countries': {'height': 550},
},
show_name=False),
self.plot,
)
CovidPlotter().panel()
Continuing this process we created several visualizations that are all tied into a single Panel app. We exported the code into external Python files that can then be simply imported and executed in a Notebook:
from covid_dashboard import CovidDashboard
Complete Panel App¶
CovidDashboard().panel()
Converting to a Stand-alone Bokeh App¶
When running a notebook locally you can launch a stand-alone Bokeh app right from the notebook by adding a .show() to any Panel output.
In our case this would be:
CovidDashboard().panel().show()
Alternatively you can create a Python script, main.py, similar to the following:
# filename: main.py
from covid_dashboard import CovidDashboard
cd = CovidDashboard().panel()
cd.servable()
If the main.py script were located in a directory called bokeh_app/ then this Bokeh application could be launched from the commandline like this:
panel serve /path/to/bokeh_app/
To see this served as a Bokeh app in Binder see the following link:
Converting to a Production Tethys App¶
Converting the app to Tethys is not quite as simple as moving from the notebook to a Bokeh app, but it is still very easy and requires minimal code.
After creating a Tethys App Scaffold, then a handlers.py file needs to be added that is similar to the main.py file mentioned above. It should contain code that is something like this:
from bokeh.document import Document
from covid_dashboard import CovidDashboard
def handler(doc: Document) -> None:
cd = CovidDashboard().panel()
cd.server_doc(doc)
Next the default home controller in the controllers.py file should be modified to contain the following:
def home(request):
"""
Controller for the app home page.
"""
script = server_document(request.get_full_path())
context = {
'script': script,
}
return render(request, 'covid/home.html', context)
Finally, the handler needs to be registered with the default home UrlMap in app.py:
UrlMap(
name='home',
url='covid',
controller='covid.controllers.home',
handler='covid.handlers.handler',
handler_type='bokeh',
)
With those changes the app will then be integrated into a Tethys app. Additional changes can then be made to take advantage of the Tethys framework.