Interactive 3D and 2D Visualization in Jupyter

TriPython March 2018 Meeting

Matthew McCormick, PhD

Kitware, Inc


Where have we come from? Where are we going?

What is the problem we are solving?

What technical approaches are required for this problem?

How do I use this tool?

What are the next steps?

Where have we come from?

Where are we going?

with interactive scientific computing in the SciPy ecosystem

IPython Shell

Original author: Fernando PĂ©rez

Image source

Initial release: 2001

IPython Shell

IPython 0.7.2 released June 6, 2006

[Image source](

  • Interactive: tab completion
  • Accessible: Matlab $\rightarrow$ Scientific Python
  • Brings powerful programming tools together: matplotlib integration

IPython Notebook

IPython 0.12.0 released December 18, 2011

[Image source](

  • Interactive: creates a tool for open, collaborative, reproducible scientific computing
  • Accessible: Terminal $\rightarrow$ Web browser
  • Brings powerful programming tools together: Literate programming: Markdown + $\LaTeX$ prose, IPython shell code cells, and matplotlib visualizations

Jupyter Notebook

Fernando announces Project Jupyter at SciPy 2014

[Image source](

  • Interactive: Adds a file browser, browser-based terminal, and browser-based text editor
  • Accessible: Open standards for interative computing to enabled customized applications: Notebook Document Format, Interactive Computing Protocol, The Kernel
  • Brings powerful programming tools together: Expands support for programming language kernels other than Python


"JupyterLab is Ready for Users" - February 20, 2018

[Image source](

  • Interactive: Browser-based window management, customized viewers for different file types
  • Accessible: Drag and drop cells, easy deploy with Docker-based JupyterHub
  • Brings powerful programming tools together: Made with community-developed extensions in mind

The Future

[Image source](

"To start developing a JupyterLab extension, see the JupyterLab Extension Developer Guide and the TypeScript or JavaScript extension templates. JupyterLab itself is co-developed on top of PhosphorJS, a new Javascript library for building extensible, high-performance, desktop-style web applications. We use modern JavaScript technologies such as TypeScript, React, Lerna, Yarn, and webpack."

  • Interactive: Browser-based application
  • Accessible: Zero-install, works on all platforms
  • Brings powerful programming tools together: JavaScript, WebGL, WebAssembly

What is the problem we are solving?

Interactive visualization to support multi-dimensional spatial analysis


A Jupyter / Leaflet bridge enabling interactive maps in the Jupyter notebook.

[Image source](


GeoNotebook is an application that provides client/server environment with interactive visualization and analysis capabilities using Jupyter, GeoJS and other open source tools.

[Image source](


3d plotting for Python in the Jupyter notebook based on IPython widgets using WebGL.

[Image source](


A Python / ThreeJS bridge utilizing the Jupyter widget infrastructure.

[Image source](


These widgets are designed to support spatial analysis with the Insight Toolkit (ITK), but they also work with other spatial analysis tools in the scientific Python ecosystem.

These widgets are built on itk.js and vtk.js.

  • 2D or 3D images
  • Excellent volume rendering
  • Slicing
  • Data probe
  • Histogram and opacity transfer function editor
In [2]:
from itkwidgets import view
import itk
PixelType = itk.ctype('float')
image = itk.imread(brainFileName, PixelType)
In [3]:
# Smooth the image
smoother = itk.CurvatureFlowImageFilter.New(image)
In [4]:
confidence_connected = itk.ConfidenceConnectedImageFilter.New(smoother.GetOutput())

Dimension = image.GetImageDimension()
SeedType = itk.Index[Dimension]
seed1 = SeedType()
seed1[0] = 118
seed1[1] = 133
seed1[2] = 92

seed2 = SeedType()
seed2[0] = 63
seed2[1] = 135
seed2[2] = 94

seed3 = SeedType()
seed3[0] = 63
seed3[1] = 157
seed3[2] = 90

seed4 = SeedType()
seed4[0] = 111
seed4[1] = 150
seed4[2] = 90

seed5 = SeedType()
seed5[0] = 111
seed5[1] = 50
seed5[2] = 88
In [5]:

What technical approaches are required for this problem?

Multiple Languages

Attention to Performance

  • Browser memory limitations? No!
  • Browser compute limitations? If so, not for long!
  • Data transfer times? Yes!

How do I use this tool?

Window / Level

[Image source](

Opacity Transfer Function

  • Revelant to volume rendering
  • Scalar intensity opacity
  • Gradient-based opacity (shading)

What are the next steps?

Impact: Addressing Heart Disease

American Heart Association, Precision Medicine Platform

[Image source](

Impact: Smart, Data-driven Financial Investments

Satellite Image Analysis

[Image source](

Enable People to Unlock the Power of Math, Science, and Computing

Next Up for itk-jupyter-widgets

  • Support image types other than itk.Image, NumPy Array, e.g. OpenCV Mat, ImageJ2 images, vtkImageData, ...
  • Rendering meshes and point clouds
  • Compression and dynamic downsampling
  • Documentation
  • Embedding
  • JupyterLab Support
  • Interactive
  • Accessible
  • Brings powerful programming tools people together

Email: [email protected]