Interactive Computing with Jupyter and Almond

space/shift+space to navigate slides

ctrl+enter to run code

up/down to select the previous/next code cell

optimized for full-screen mode

Sören Brunk
@soebrunk

How Do We learn?

Exploration is a Feedback Loop

Interactive Computing

An Interactive computation is a persistent computer program that runs with a "human in the loop," where the primary mode of interaction is through the human interactively writing/running blocks of code and looking at the result. — Brian Granger (co-founder of Project Jupyter)

The Scala 2 REPL

Ammonite

Limitations of the REPL Design

  • Optimized for exploration from scratch
    • Write code and see outputs fast (feedback loop)
  • Not ideal for reading, copying or changing code
    • Interleaved inputs and outputs
    • History cluttered with refinement attempts
    • Dependent expressions
  • Sharing is difficult
    • Not persistent except history
  • Observe and replicate phases not part the of REPL design

Learning with the REPL

Worksheets

A worksheet is a Scala file that is evaluated on save, and the result of each expression is shown in a column to the right of your program. Worksheets are like a REPL session on steroids, and enjoy 1st class editor support: completion, hyperlinking, interactive errors-as-you-type, etc.

Source: Dotty documentation

Online Worksheets

Learning with Worksheets

A More Integrated Experience?

Tour of Scala

Higher-Order-Functions

Higher order functions take other functions as parameters or return other functions ...

One of the most common examples is the higher-order function map which is available for collections in Scala.

In [ ]:
// Use ctrl+enter to run the cell
val salaries = Seq(20000, 70000, 40000)
val doubleSalary = (x: Int) => x * 2
val newSalaries = salaries.map(doubleSalary)
salaries.map(x => x *3)

doubleSalary is a function which takes a single Int, x, and returns x * 2. In general, the tuple on the left of the arrow => is a parameter list and the value of the expression on the right is what gets returned. On line 3, the function doubleSalary gets applied to each element in the list of salaries.

How?

  • We're actually in a Jupyter notebook!
  • It's just shown as a slideshow (using the RISE plugin)
  • Usually, it looks more like this:

Notebook Basics

  • Jupyter notebooks are interactive, web based documents
  • They are made of cells
  • Two cell types
    • Documentation
    • Code

Documentation Cells

  • All the usual Markdown formatting options
  • Images Scala
  • Code fences
    val x = 23
    
  • LaTeX equations $$e^x=\sum_{i=0}^\infty \frac{1}{i!}x^i$$
  • We can edit them right in the browser!

Double click to enter edit mode, then hit ctrl+enter to return to the rendered view

Code Cells

  • Write code
  • Evaluate code
In [ ]:
case class Person(name: String, age: Int)

val alice = Person("alice", 42)
val bob = Person("bob", 43)

Seq(alice, bob).map(_.name.capitalize)

Mix and Match

  • We can have any number of cells in a notebook
  • We can freely mix code cells with documentations cells
In [ ]:
"I'm a code cell, evaluate me!"
In [ ]:
println("I'm so effectful!")
  • We can even put cells side-by-side
    (with the help of a plugin)
In [ ]:
"I'm smaller, but you can still evaluate me."

Notebooks Integrate Rich Documentation with Runnable Code

Notebooks are Worksheets in a REPL

Notebooks are Worksheets in a REPL

DEMO

Exploring Data with the REPL

Exploring Data with Notebooks

In [ ]:
interp.repositories() ++= Seq(coursier.maven.MavenRepository("https://jitpack.io"))
import $ivy.`org.apache.spark::spark-sql:2.4.3` // Or use any other 2.x version here
import $ivy.`sh.almond::almond-spark:0.5.0`
import org.apache.spark.sql._, org.apache.log4j.{Level, Logger}
Logger.getLogger("org").setLevel(Level.OFF)

val spark = {
  NotebookSparkSession.builder()
    .progress(false)
    .master("local[*]")
    .getOrCreate()
}
def sc = spark.sparkContext

Exploring Data with Notebooks

In [ ]:
import spark.implicits._
import $file.SparkHelpers, SparkHelpers._

val titanic = spark.read
  .format("csv")
  .option("inferSchema", "true").option("header", "true")
  .load("titanic.csv")

titanic.showHTML()

Exploring Data with Notebooks

In [ ]:
import $ivy.`org.vegas-viz::vegas-spark:0.3.12-SNAPSHOT`
import vegas._, vegas.data.External._, vegas.sparkExt._
Vegas().
  withDataFrame(titanic).
  mark(vegas.Bar).
  encodeY("*", aggregate=AggOps.Count, axis=vegas.Axis(title="Number of People", grid=false)).
  encodeColumn("Pclass", Ord, scale=Scale(padding=16.0), axis=vegas.Axis(orient=Orient.Bottom, axisWidth=1.0, offset= -8.0)).
  encodeX("Survived", Nominal, scale=Scale(bandSize = 20.0), hideAxis=true).
  encodeColor("Survived", Nominal, scale=Scale(rangeNominals=List("red", "green"))).
  configFacet(cell=CellConfig(strokeWidth = 0)).configCell(height=400).show

Exploring Data with Notebooks

In [ ]:
import $ivy.`org.plotly-scala::plotly-almond:0.7.0`
import plotly._, plotly.element._, plotly.layout._, plotly.Almond._

val trace1 = plotly.Bar(
  Seq("giraffes", "orangutans", "monkeys"), Seq(20, 14, 23), name = "SF Zoo"
)
val trace2 = plotly.Bar(
  Seq("giraffes", "orangutans", "monkeys"), Seq(15, 18, 29), name = "LA Zoo"
)
val data = Seq(trace1, trace2)
val layout = Layout(barmode = BarMode.Group)

plot(data, layout)

But we aren't restricted to visualizing data. We can also visualize code.

Visualizing Code with Notebooks

In [ ]:
import $ivy.`io.github.stanch::reftree:1.4.0`
import reftree.render._, reftree.diagram._, reftree.contrib.SimplifiedInstances.string

case class Person(name: String, age: Int)
val people = List(Person("Alice", 29), Person("Bob", 25))

val renderer = Renderer(renderingOptions = RenderingOptions(density = 80))
renderer.render("example", Diagram.sourceCodeCaption(people))
Image.fromFile("example.png")

Notebooks Enable Rich Embedded Output

Project Jupyter

Jupyter History

2001: IPython Shell

2011: IPython Notebook

2014: Project Jupyter (support for other languages besides Python)

Julia
Python
R

2018: JupyterLab (next generation UI)

High-Level Jupyter Architecture

Jupyter components

(source: Jupyter documentation)

Jupyter Frontends

  • Classic Notebook - first Jupyter UI, still widely used
  • JupyterLab - modern frontend implementation
  • nteract - minimalistic desktop based frontend
  • Hydrogen atom plugin - worksheet like with Jupyter backend
  • IntelliJ and VS Code also have limited notebook support
    • Currently restricted to Python

JupyterLab

  • Modernized UI
  • Modular, component based, modern web technologies
  • More than notebooks:
    • File manager
    • Editor
    • Terminal
    • Plugins
  • Close to a 1.0 release

JupyterLab

The Jupyter Protocol

Kernels

  • Responsible for running code
  • Don't know anything about notebooks
  • Can be implemented in any language
    • Have to support the Jupyter protocol
  • ~ 100 different Kernels available

The almond Scala Kernel

Ammonite Niceties

  • Dynamic dependency resolution via Coursier
In [ ]:
import $ivy.`org.typelevel::squants:1.4.0`
import squants.energy.Kilowatts, squants.time.Hours

val load = Kilowatts(1.2)
val time = Hours(2)
val energyUsed = load * time
  • Syntax highlighting and pretty printing
In [ ]:
Seq.fill(3)(Seq.fill(5)(f"${scala.util.Random.nextFloat}%1.4f"))

Not A Full Blown IDE - But

  • Autocompletion for regular code and ivy imports
  • Type hints & metabrowse via shift + tab (work in progress)
In [ ]:
import $ivy.`org.typelevel::squants:` // put the cursor after the last colon and press shift, select 1.4.0

Seq("alice", "bob", "charlie").map(name => name.) // put the cursor after name. and press shift, select capitalize 
// Run the cell, then put the cursor on map and press shift+tab multiple times until you get a type hint at the bottom
// Click on the link int the type hint for a metabrowse window

Using the Ammonite API

In [ ]:
repl.history.takeRight(3).toList
In [ ]:
interp.repositories().last

Rich Outputs with the almond Jupyter API

In [ ]:
import almond.display._

Html("Some <b>bold</b> text")
In [ ]:
Svg("""<svg height="250" xmlns="http://www.w3.org/2000/svg">
        <rect x="25" y="25" width="200" height="200" fill="lime" stroke-width="4" stroke="pink" />
        <circle cx="125" cy="125" r="75" fill="orange" />
       </svg>""")
In [ ]:
Image.from(url = "http://localhost:8888/static/base/images/logo.png")
In [ ]:
Latex("""$$e^x=\sum_{i=0}^\infty \frac{1}{i!}x^i$$""")
In [ ]:
Javascript("alert('Hello')") // JS execution is restricted in newer frontends

Finally some Progress...

In [ ]:
val handle = ProgressBar(10,100)
  .withHtmlWidth("100%")
  .withLabel("Overall Progress")
In [ ]:
for {i <- 1 to 100} {
  handle.withProgress(i).update()
  Thread.sleep(30)
}

Reading User Input

In [ ]:
val result = Input().withPrompt(">>> ").request()

Updating Vars

In [ ]:
import scala.concurrent.Future
implicit val ec = scala.concurrent.ExecutionContext.global

val f = Future {
  Thread.sleep(10000)
  "finished"
}

val v = "a val"
var x = "a var that we'll update"
var y = "another var"
In [ ]:
x = "updated x"

Using the almond APIs for Library Integrations

build.sbt of your library:

libraryDependencies += ("sh.almond" % "scala-kernel-api" % "0.5.0" % Provided)
  .cross(CrossVersion.full))

Your library code:

In [ ]:
def randomEmoji()(implicit kernel: almond.api.JupyterApi): Unit = {
  val emoji = Character.toChars(scala.util.Random.nextInt(0x1F64F - 0x1F600) + 0x1F600).mkString
  kernel.publish.html(s"""<p style="font-size:4em; text-align: center">$emoji</p>""")}

Some notebook:

In [ ]:
// import $ivy.`my:library:1.0`
randomEmoji()

Teaching with Notebooks

How To Make Your Existing Documentation Interactive

  • Many tutorials are already Markdown with examples in code fences
    • Often even checked via tut or mdoc
  • Notebook contents are very similar, just encoded differently (JSON)

Share Your Runnable Notebooks With Binder

DEMO

Contributors Welcome!

  • We'd love to see
    • More library/framework integrations
    • More example notebooks
    • Better IDE integration
    • A logo :)
  • Interested?
    • ⇒ Talk to Alex (@alxarchambault) or me (@soebrunk)
    • ⇒ Join our Gitter channel: https://gitter.im/almond-sh/almond
    • ⇒ Or check out the almond repo on https://github.com/almond-sh/almond

Recap

  • Jupyter notebooks

    • Integrate documentation, runnable code and rich output in a single document
    • Combine REPLs and worksheets
  • almond combines the power of Jupyter and Ammonite

    • Giving us first class interactive computing support in Scala

Questions?

Sören Brunk
@soebrunk