Embedding Abjad MIDI Output in IPython Notebooks

Recently, I was really impressed by Tiago Antao's Abjad-IPython integration which renders Abjad code into PNGs of the Lilypond output. It was recently added to the Abjad repository which is really exciting, you can see an example of this integration below. To further his work and to satisfy my audio interests, I explored two methods of creating dynamically rendered audio of Abjad notation within IPython notebooks.

Demonstrative Abjad Code

In [8]:
from abjad import *
%load_ext abjad.ext.ipython

pitch_numbers = 4 * [0, 2, 4, 9, 7]
duration = Duration(1, 8)
notes = scoretools.make_notes(pitch_numbers, duration)
staff = Staff(notes)
show(staff)

Attempt #1: <embed> Tag

This is the first method I explored, which was extremely common back in the early 2000s but I knew it would be easy to implement. It relies on the <embed> tag which expects browser plugins to be able to handle the content within the tags. This style of integration is being phased out, as MIDI playback has been removed from the Safari-Quicktime plugin since 10.8. Unfortunately the Web MIDI API is too young for widely accepted usage, so nothing is replacing this old functionality.

In [15]:
import os
import tempfile
from abjad.tools import systemtools, topleveltools

tmpdir = tempfile.mkdtemp()
agent = topleveltools.persist(staff)
result = agent.as_midi(tmpdir + os.sep + 'out.mid')
midi_file_path, format_time, render_time = result

from IPython.display import HTML
from base64 import b64encode
midi_file = open(midi_file_path, "rb").read()
midi_encoded = b64encode(midi_file)
embed_tag = '<embed alt="ERROR" type="audio/midi" src="data:audio/midi;base64,{0}">'.format(midi_encoded)
HTML(data=embed_tag)
Out[15]:

Attempt #2: Python Rendering Into <audio> Tag

This method involves rendering the MIDI file using fluidsynth, and dropping the resulting audio in an <audio> tag using the base64 encoding. For this to work you need to install fluidsynth with your familiar package manager, and Soundfont file to render the MIDI with.

In this example I am using FluidR3 General MIDI, and made the assumption that it is in the same directory as this notebook.

$ brew install fluidsynth
In [16]:
import os
import tempfile
from abjad.tools import systemtools, topleveltools

tmpdir = tempfile.mkdtemp()
agent = topleveltools.persist(staff)
result = agent.as_midi(tmpdir + os.sep + 'out.mid')
midi_file_path, format_time, render_time = result

soundfont = 'FluidR3_GM.sf2'
tmpaudio = tmpdir + os.sep + 'out.ogg'
cmd = 'fluidsynth -nli -r 48000 -o synth.cpu-cores=2 -T oga -F %s %s %s' % (tmpaudio, soundfont, midi_file_path)
result = systemtools.IOManager.spawn_subprocess(cmd)

from IPython.display import HTML
from base64 import b64encode
audio_file = open(tmpaudio, "rb").read()
audio_encoded = b64encode(audio_file)
audio_tag = '<audio controls type="audio/ogg" src="data:audio/ogg;base64,{0}">'.format(audio_encoded)
HTML(data=audio_tag)
Out[16]:

Further Thoughts

Given how quickly fluidsynth renders audio, I was quite satisfied using it as a permanent solution. Another possible alternative would be client-side rendering using something like MIDI.js, but I ran into difficulties integrating Javascript within IPython (Though I am fairly new to IPython). I think that using base64 encoding to store the file directly in the notebook is probably better than dynamically rendering content anyway.

I am going to try to clean this code up and make a nice repository for it. Thanks for reading this notebook!

Griffin Moe | griffinmoe.com | GitHub

In [ ]: