ipythonPexpect example

Adam Lyon, Fermi National Accelerator Laboratory, lyon at fnal.gov, March 2013

Updated May 2015 - The input command and prompt are not repeated in the result cell unless you set "-a".

ipythonPexpect is an IPython notebook extension that interfaces with the pexpect python module by Noah Spurrier. See https://pypi.python.org/pypi/pexpect/ and http://www.noah.org/wiki/Pexpect for detailed information on pexpect.

pexpect is a module that spawns a process that starts some application program (like a shell, R, CERN's Root, even Python). You can then feed it commands, capture the output, and wait for the next prompt. Most any command line driven application that provides a consistent prompt can be used. We'll do some examples here.

There is also a feature to lock the notebook such that all subsequent commands that you give, even without the magic, will go to the pexpect session. This is very convenient for long sessions, but may be surprising to other readers. So be sure to explain what you are doing carefully.

See https://cdcvs.fnal.gov/redmine/projects/ipython_ext/wiki/Wiki for more information including how to download this extension.

First, we load the notebook extension.

In [1]:
%load_ext ipythonPexpect
%ipythonPexpect? for help


There are some prewritten spawn scripts to start bash, R, CERN's ROOT, and python. Let's try a bash shell (note that prompt is changed to a known string).

In [2]:
Opened connection to /usr/bin/env bash
bash-3.2$ PS1='bash> '

We can now send commands to this bash session with %%P.

In [3]:
Tue May 12 00:39:20 CDT 2015

pexpect sessions are persistent, so the state is maintained between commands (unlike ! which starts a new shell on each call)

In [4]:
!export FOO="hi"
In [5]:
!echo $FOO

In [6]:
export FOO="hi"
In [7]:
echo $FOO

If you want to see output in real time, you need to set the -a option to show the command and the output.

In [8]:
%%P -a
for i in $(seq 1 5); do sleep 2; date; done
bash> for i in $(seq 1 5); do sleep 2; date; done
Tue May 12 00:39:22 CDT 2015
Tue May 12 00:39:24 CDT 2015
Tue May 12 00:39:27 CDT 2015
Tue May 12 00:39:29 CDT 2015
Tue May 12 00:39:31 CDT 2015

Note that blank lines and white space are seen

In [9]:
%%P -a
   echo "hi"
bash>    echo "hi"

When done, close the session

In [10]:
Closed connection to /usr/bin/env bash


Let's try a python session

In [11]:
Opened connection to /usr/bin/env python
Python 2.7.9 |Anaconda 2.2.0 (x86_64)| (default, Dec 15 2014, 10:37:34) 
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
In [12]:
print 'hi'
In [13]:
print 'hi'
print 'bye'

For the command below, note the two blank lines at the end (for some reason, nbviewer doesn't show it, but it's there in the actual notebook)

In [14]:
for i in range(5):
  print i
In [15]:
print 'hi'+ \
  ' adam'
hi adam
In [16]:
Closed connection to /usr/bin/env python


Try R

In [17]:
Opened connection to R

R version 3.1.3 (2015-03-09) -- "Smooth Sidewalk"
Copyright (C) 2015 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin13.4.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

In [18]:
In [19]:
a <- rnorm(20000)
[1] 0.52805098 0.34819182 1.01972016 0.74633602 0.04287176 0.58574984

You can add an image if you know the output file name.

In [20]:
%%P -f rout.png
null device

You can return values to python (you may have to slice and dice the return string) with the -e (evaluate) option.

In [21]:
pya = %%P -e mean(a)
[1] 0.009820224
In [22]:
'[1] 0.009820224'
In [23]:
Closed connection to R


Note that ROOT is a special challenge because it does syntax highlighting and coloring at its command line. This leads to lots of ugly ANSI codes littering the output. The %pexpect_spawn_root magic writes a .rootrc file to the current directory that will turn Root to turn off this highlighting.

In [24]:
A .rootrc file is automatically written to turn off root prompt syntax coloring.
But it cannot be written because one already exists. Continuing assuming
.rootrc turns off syntax coloring.
Opened connection to root -b
  *                                         *
  *        W E L C O M E  to  R O O T       *
  *                                         *
  *   Version   5.34/25   12 January 2015   *
  *                                         *
  *  You are welcome to visit our Web site  *
  *          http://root.cern.ch            *
  *                                         *

ROOT 5.34/25 ([email protected], Apr 17 2015, 12:16:00 on macosx64)

CINT/ROOT C/C++ Interpreter version 5.18.00, July 2, 2010
Type ? for help. Commands must be C++ statements.
Enclose multiple statements between { }.
root [0] 

We're going to try the phase space tutorial

In [25]:

I'm getting tired of entering %%P. I'm going to lock the notebook in Root mode. Make sure you know what you are doing here. When I'm done, I have to %pexpect_unlock in order for the notebook to process IPython again.

In [26]:
WARNING: All future cell execution will be processed through pexpect!
To return to IPython, issue %pexpect_unlock

Continuing onward...

In [27]:
TLorentzVector target(0.0, 0.0, 0.0, 0.938)
TLorentzVector beam(0.0, 0.0, 0.65, 0.65)
TLorentzVector W = beam + target
In [28]:
Double_t masses[3] = { 0.938, 0.139, 0.139 }
In [29]:
In [30]:
TGenPhaseSpace event
In [31]:
event.SetDecay(W, 3, masses)
In [32]:
 TH2F *h2 = new TH2F("h2","h2", 50,1.1,1.8, 50,1.1,1.8);

For some reason, if you don't put blocks of code within braces, you get syntax coloring again which confuses the code that removes the commands from the output.

In [33]:
  for (Int_t n=0;n<100000;n++) {
      Double_t weight = event.Generate();

      TLorentzVector *pProton = event.GetDecay(0);

      TLorentzVector *pPip    = event.GetDecay(1);
      TLorentzVector *pPim    = event.GetDecay(2);

      TLorentzVector pPPip = *pProton + *pPip;
      TLorentzVector pPPim = *pProton + *pPim;

      h2->Fill(pPPip.M2() ,pPPim.M2() ,weight);
In [34]:
(class TH2F*)0x7fc81bf25ae0
In [35]:
Info in <TCanvas::MakeDefCanvas>:  created default TCanvas with name c1

Event though I'm in "locked" mode, I can still issue %%P if I need to give it options.

In [36]:
%%P -f root.png
Info in <TCanvas::Print>: png file root.png has been created
In [37]:
Notebook will use IPython
In [38]:
Closed connection to root -b

Your own application

Use the %pexpect_spawn command and be sure to ask the system for help for the options.

In [39]:

Let's try to connect to ruby. Note that irb immediately shows the prompt with no preamble. But the prompt is pretty involved, so we will leave out the \r\n.

In [40]:
%pexpect_spawn -p 'irb.+> ' -c 'irb.+\* ' -e irb
Opened connection to irb
In [41]:
=> 4
In [42]:
=> nil
In [43]:
2 + 
=> 4
In [44]:
%%P -a
2 +
irb(main):005:0> 2 +
irb(main):006:0* 2
=> 4
In [45]:
$i = 0
$num = 5

until $i > $num  do
   puts("Inside the loop i = #$i" )
   $i +=1;
=> 0
=> 5
Inside the loop i = 0
Inside the loop i = 1
Inside the loop i = 2
Inside the loop i = 3
Inside the loop i = 4
Inside the loop i = 5
=> nil
In [46]:
Closed connection to irb