# Interactive Computing with Jupyter and Almond

space/shift+space to navigate slides

ctrl+enter to run code

up/down to select the previous/next code cell

optimized for full-screen mode

Sören Brunk
@soebrunk

# Interactive Computing¶

An Interactive computation is a persistent computer program that runs with a "human in the loop," where the primary mode of interaction is through the human interactively writing/running blocks of code and looking at the result. — Brian Granger (co-founder of Project Jupyter)

# Limitations of the REPL Design¶

• Optimized for exploration from scratch
• Write code and see outputs fast (feedback loop)
• Not ideal for reading, copying or changing code
• Interleaved inputs and outputs
• History cluttered with refinement attempts
• Dependent expressions
• Sharing is difficult
• Not persistent except history
• Observe and replicate phases not part the of REPL design

# Worksheets¶

A worksheet is a Scala file that is evaluated on save, and the result of each expression is shown in a column to the right of your program. Worksheets are like a REPL session on steroids, and enjoy 1st class editor support: completion, hyperlinking, interactive errors-as-you-type, etc.

Source: Dotty documentation

# A More Integrated Experience?¶

Tour of Scala

Higher-Order-Functions

Higher order functions take other functions as parameters or return other functions ...

One of the most common examples is the higher-order function map which is available for collections in Scala.

In [ ]:
// Use ctrl+enter to run the cell
val salaries = Seq(20000, 70000, 40000)
val doubleSalary = (x: Int) => x * 2
val newSalaries = salaries.map(doubleSalary)
salaries.map(x => x *3)


doubleSalary is a function which takes a single Int, x, and returns x * 2. In general, the tuple on the left of the arrow => is a parameter list and the value of the expression on the right is what gets returned. On line 3, the function doubleSalary gets applied to each element in the list of salaries.

# How?

• We're actually in a Jupyter notebook!
• It's just shown as a slideshow (using the RISE plugin)
• Usually, it looks more like this:

# Notebook Basics

• Jupyter notebooks are interactive, web based documents
• They are made of cells
• Two cell types
• Documentation
• Code

# Documentation Cells¶

• All the usual Markdown formatting options
• Images
• Code fences
val x = 23

• LaTeX equations $$e^x=\sum_{i=0}^\infty \frac{1}{i!}x^i$$
• We can edit them right in the browser!

Double click to enter edit mode, then hit ctrl+enter to return to the rendered view

# Code Cells¶

• Write code
• Evaluate code
In [ ]:
case class Person(name: String, age: Int)

val alice = Person("alice", 42)
val bob = Person("bob", 43)

Seq(alice, bob).map(_.name.capitalize)


# Mix and Match¶

• We can have any number of cells in a notebook
• We can freely mix code cells with documentations cells
In [ ]:
"I'm a code cell, evaluate me!"

In [ ]:
println("I'm so effectful!")

• We can even put cells side-by-side
(with the help of a plugin)
In [ ]:
"I'm smaller, but you can still evaluate me."


# Exploring Data with Notebooks¶

In [ ]:
interp.repositories() ++= Seq(coursier.maven.MavenRepository("https://jitpack.io"))
import $ivy.org.apache.spark::spark-sql:2.4.3 // Or use any other 2.x version here import$ivy.sh.almond::almond-spark:0.5.0
import org.apache.spark.sql._, org.apache.log4j.{Level, Logger}
Logger.getLogger("org").setLevel(Level.OFF)

val spark = {
NotebookSparkSession.builder()
.progress(false)
.master("local[*]")
.getOrCreate()
}
def sc = spark.sparkContext


# Exploring Data with Notebooks¶

In [ ]:
import spark.implicits._
import $file.SparkHelpers, SparkHelpers._ val titanic = spark.read .format("csv") .option("inferSchema", "true").option("header", "true") .load("titanic.csv") titanic.showHTML()  # Exploring Data with Notebooks¶ In [ ]: import$ivy.org.vegas-viz::vegas-spark:0.3.12-SNAPSHOT
import vegas._, vegas.data.External._, vegas.sparkExt._
Vegas().
withDataFrame(titanic).
mark(vegas.Bar).
encodeY("*", aggregate=AggOps.Count, axis=vegas.Axis(title="Number of People", grid=false)).
encodeColumn("Pclass", Ord, scale=Scale(padding=16.0), axis=vegas.Axis(orient=Orient.Bottom, axisWidth=1.0, offset= -8.0)).
encodeX("Survived", Nominal, scale=Scale(bandSize = 20.0), hideAxis=true).
encodeColor("Survived", Nominal, scale=Scale(rangeNominals=List("red", "green"))).
configFacet(cell=CellConfig(strokeWidth = 0)).configCell(height=400).show


In [ ]:
import $ivy.org.plotly-scala::plotly-almond:0.7.0 import plotly._, plotly.element._, plotly.layout._, plotly.Almond._ val trace1 = plotly.Bar( Seq("giraffes", "orangutans", "monkeys"), Seq(20, 14, 23), name = "SF Zoo" ) val trace2 = plotly.Bar( Seq("giraffes", "orangutans", "monkeys"), Seq(15, 18, 29), name = "LA Zoo" ) val data = Seq(trace1, trace2) val layout = Layout(barmode = BarMode.Group) plot(data, layout)  But we aren't restricted to visualizing data. We can also visualize code. # Visualizing Code with Notebooks¶ In [ ]: import$ivy.io.github.stanch::reftree:1.4.0
import reftree.render._, reftree.diagram._, reftree.contrib.SimplifiedInstances.string

case class Person(name: String, age: Int)
val people = List(Person("Alice", 29), Person("Bob", 25))

val renderer = Renderer(renderingOptions = RenderingOptions(density = 80))
renderer.render("example", Diagram.sourceCodeCaption(people))
Image.fromFile("example.png")


# Jupyter History¶

2001: IPython Shell

2011: IPython Notebook

2014: Project Jupyter (support for other languages besides Python)

2018: JupyterLab (next generation UI)

# High-Level Jupyter Architecture

(source: Jupyter documentation)

# Jupyter Frontends¶

• Classic Notebook - first Jupyter UI, still widely used
• JupyterLab - modern frontend implementation
• nteract - minimalistic desktop based frontend
• Hydrogen atom plugin - worksheet like with Jupyter backend
• IntelliJ and VS Code also have limited notebook support
• Currently restricted to Python

# JupyterLab

• Modernized UI
• Modular, component based, modern web technologies
• More than notebooks:
• File manager
• Editor
• Terminal
• Plugins
• Close to a 1.0 release

# Kernels¶

• Responsible for running code
• Don't know anything about notebooks
• Can be implemented in any language
• Have to support the Jupyter protocol
• ~ 100 different Kernels available

# Ammonite Niceties¶

• Dynamic dependency resolution via Coursier
In [ ]:
import $ivy.org.typelevel::squants:1.4.0 import squants.energy.Kilowatts, squants.time.Hours val load = Kilowatts(1.2) val time = Hours(2) val energyUsed = load * time  • Syntax highlighting and pretty printing In [ ]: Seq.fill(3)(Seq.fill(5)(f"${scala.util.Random.nextFloat}%1.4f"))


# Not A Full Blown IDE - But¶

• Autocompletion for regular code and ivy imports
• Type hints & metabrowse via shift + tab (work in progress)
In [ ]:
import ivy.org.typelevel::squants: // put the cursor after the last colon and press shift, select 1.4.0 Seq("alice", "bob", "charlie").map(name => name.) // put the cursor after name. and press shift, select capitalize // Run the cell, then put the cursor on map and press shift+tab multiple times until you get a type hint at the bottom // Click on the link int the type hint for a metabrowse window  # Using the Ammonite API¶ In [ ]: repl.history.takeRight(3).toList  In [ ]: interp.repositories().last  # Rich Outputs with the almond Jupyter API¶ In [ ]: import almond.display._ Html("Some <b>bold</b> text")  In [ ]: Svg("""<svg height="250" xmlns="http://www.w3.org/2000/svg"> <rect x="25" y="25" width="200" height="200" fill="lime" stroke-width="4" stroke="pink" /> <circle cx="125" cy="125" r="75" fill="orange" /> </svg>""")  In [ ]: Image.from(url = "http://localhost:8888/static/base/images/logo.png")  In [ ]: Latex("""$$e^x=\sum_{i=0}^\infty \frac{1}{i!}x^i$$""")  In [ ]: Javascript("alert('Hello')") // JS execution is restricted in newer frontends  # Finally some Progress...¶ In [ ]: val handle = ProgressBar(10,100) .withHtmlWidth("100%") .withLabel("Overall Progress")  In [ ]: for {i <- 1 to 100} { handle.withProgress(i).update() Thread.sleep(30) }  # Reading User Input¶ In [ ]: val result = Input().withPrompt(">>> ").request()  # Updating Vars¶ In [ ]: import scala.concurrent.Future implicit val ec = scala.concurrent.ExecutionContext.global val f = Future { Thread.sleep(10000) "finished" } val v = "a val" var x = "a var that we'll update" var y = "another var"  In [ ]: x = "updated x"  # Using the almond APIs for Library Integrations¶ build.sbt of your library: libraryDependencies += ("sh.almond" % "scala-kernel-api" % "0.5.0" % Provided) .cross(CrossVersion.full))  Your library code: In [ ]: def randomEmoji()(implicit kernel: almond.api.JupyterApi): Unit = { val emoji = Character.toChars(scala.util.Random.nextInt(0x1F64F - 0x1F600) + 0x1F600).mkString kernel.publish.html(s"""<p style="font-size:4em; text-align: center">emoji</p>""")}


Some notebook:

In [ ]:
// import \$ivy.my:library:1.0
randomEmoji()


# How To Make Your Existing Documentation Interactive¶

• Many tutorials are already Markdown with examples in code fences
• Often even checked via tut or mdoc
• Notebook contents are very similar, just encoded differently (JSON)

# Contributors Welcome!¶

• We'd love to see
• More library/framework integrations
• More example notebooks
• Better IDE integration
• A logo :)
• Interested?
• ⇒ Talk to Alex (@alxarchambault) or me (@soebrunk)
• ⇒ Or check out the almond repo on https://github.com/almond-sh/almond

# Recap¶

• Jupyter notebooks

• Integrate documentation, runnable code and rich output in a single document
• Combine REPLs and worksheets
• almond combines the power of Jupyter and Ammonite

• Giving us first class interactive computing support in Scala

Sören Brunk
@soebrunk