This notebook is part of PyBroMo a python-based single-molecule Brownian motion diffusion simulator that simulates confocal smFRET experiments. You can find the full list of notebooks in Usage Examples.
In this notebooks we run a a brownian motion simulation on a cluster. The output of the simulation is the emission rate as a function of time.
In the next notebook we will use this data to generate timestamps.
Follow the steps in HOWTO CLUSTER SETUP.txt to configure and start an IPython cluster. There is another txt file explaining how to manage remotely windows machines (so you can start the enigines without walking in front of the remote PC).
After the cluster is started, we can proceed running the inizialization script to load the PyBroMo software:
%run -i load_bromo.py
/home/anto/Documents/ucla/src/brownian
and loading some IPython functions to manage the cluster:
from IPython.parallel import Client
from IPython.utils.path import get_ipython_dir
ipython_dir = get_ipython_dir()
PROFILE = 'parallel'
rc = Client(ipython_dir+'/profile_%s/security/ipcontroller-client.json' %\
PROFILE)
dview = rc[:]
dview.block = True
rc.ids
[4, 5, 6, 7]
If the previous cell does not give an error should return a list of integers that are the ID of all the running engines (either remote or local).
We are now ready to run the simulation in parallel.
To start a simulation we must load the functions and define the problem parameters. Some paramenters are already set in brownian.py
but can be overwritten:
%run -i brownian.py
# Simulation time step
t_step = 0.5e-6 # seconds
# Number of samples to simulate
t_max = 10 # seconds
# Particles definition
P = gen_particles(30, box)
# Object containing the simulation (parameters, functions, results)
S = ParticlesSimulation(D=D, t_step=t_step, t_max=t_max, particles=P,
box=box, psf=psf, ID=0)
We can check the simulation parameters:
S
Box: X 8.0um, Y 8.0um, Z 12.0um D 1.2e-11, #Particles 30, t_step 0.5us, t_max 10.0s EID_ID 0 0
Or have a compact string representation:
S.compact_name()
'D1.2e-11_30P_64pM_step0.5us_t_max10.0s_ID0-0'
Before running the simulation check how much RAM is requires with the current parameters:
S.print_RAM()
Number of particles: 30 Number of time steps: 20000000 Emission array size: 152.0 MB (total_emission=True) Emission array size: 4560.0 MB (total_emission=False) Position array size: 13680.0 MB
From the numbers above we see that the RAM usage can easily diverge. For long simulations we can save RAM discarding the trajectories and saving only the total emission (one array for all the particles).
With this configuration a standard 4-core PC with 8GB of RAM and runing 4 engines,
can easily perform 4 10s simulations in parallel, in about 8 minutes.
In this way a single simulation run will generate 40s of simulated emission.
In order to have longer times you can repeat the simulation with the same parameters
but a different ID
. The different IDs can be later composed when the timestamps
are generated. In this way we can simulate any desired duration of an experiment
making an efficient use of the hardware (we use all the cores).
Prepare the engines (change folder, define a unique eid per engine, load functions):
%px %reset
# Push a variable with the engine ID to each engine
dview.scatter('eid', rc.ids, flatten=True)
%px eid
Out[4:1308]: 4
Out[5:1308]: 5
Out[6:1308]: 6
Out[7:1308]: 7
Define here the folder where you put the PyBroMo software on the remote machines We can define different paths under Linux/Mac ('posix') or under Windows ('nt'):
%%px
import os
if os.name == 'posix':
BROWN_DIR = "/home/anto/Documents/ucla/src/brownian/"
elif os.name == 'nt':
BROWN_DIR = r"C:/Data/Antonio/software/Dropbox/brownian/"
%%px
%cd $BROWN_DIR
[stdout:4] C:\Data\Antonio\software\Dropbox\brownian [stdout:5] C:\Data\Antonio\software\Dropbox\brownian [stdout:6] C:\Data\Antonio\software\Dropbox\brownian [stdout:7] C:\Data\Antonio\software\Dropbox\brownian
%px run -i brownian.py
Now push the object S
to the enignes so they can run the simulation:
dview.push(dict(S=S))
[None, None, None, None]
%px S.EID = eid
%px S
Out[4:1313]:
Box: X 8.0um, Y 8.0um, Z 12.0um
D 1.2e-11, #Particles 30, t_step 0.5us, t_max 10.0s EID_ID 4 0
Out[5:1313]:
Box: X 8.0um, Y 8.0um, Z 12.0um
D 1.2e-11, #Particles 30, t_step 0.5us, t_max 10.0s EID_ID 5 0
Out[6:1313]:
Box: X 8.0um, Y 8.0um, Z 12.0um
D 1.2e-11, #Particles 30, t_step 0.5us, t_max 10.0s EID_ID 6 0
Out[7:1313]:
Box: X 8.0um, Y 8.0um, Z 12.0um
D 1.2e-11, #Particles 30, t_step 0.5us, t_max 10.0s EID_ID 7 0
Run the brownian motion simulation on all the engines:
%px S.sim_motion_em()
[stdout:0] [1156] Simulating particle 0 [1156] Simulating particle 1 [1156] Simulating particle 2 [1156] Simulating particle 3 [1156] Simulating particle 4 [1156] Simulating particle 5 [1156] Simulating particle 6 [1156] Simulating particle 7 [1156] Simulating particle 8 [1156] Simulating particle 9 [1156] Simulating particle 10 [1156] Simulating particle 11 [1156] Simulating particle 12 [1156] Simulating particle 13 [1156] Simulating particle 14 [1156] Simulating particle 15 [1156] Simulating particle 16 [1156] Simulating particle 17 [1156] Simulating particle 18 [1156] Simulating particle 19 [1156] Simulating particle 20 [1156] Simulating particle 21 [1156] Simulating particle 22 [1156] Simulating particle 23 [1156] Simulating particle 24 [1156] Simulating particle 25 [1156] Simulating particle 26 [1156] Simulating particle 27 [1156] Simulating particle 28 [1156] Simulating particle 29 [stdout:1] [2568] Simulating particle 0 [2568] Simulating particle 1 [2568] Simulating particle 2 [2568] Simulating particle 3 [2568] Simulating particle 4 [2568] Simulating particle 5 [2568] Simulating particle 6 [2568] Simulating particle 7 [2568] Simulating particle 8 [2568] Simulating particle 9 [2568] Simulating particle 10 [2568] Simulating particle 11 [2568] Simulating particle 12 [2568] Simulating particle 13 [2568] Simulating particle 14 [2568] Simulating particle 15 [2568] Simulating particle 16 [2568] Simulating particle 17 [2568] Simulating particle 18 [2568] Simulating particle 19 [2568] Simulating particle 20 [2568] Simulating particle 21 [2568] Simulating particle 22 [2568] Simulating particle 23 [2568] Simulating particle 24 [2568] Simulating particle 25 [2568] Simulating particle 26 [2568] Simulating particle 27 [2568] Simulating particle 28 [2568] Simulating particle 29 [stdout:2] [4716] Simulating particle 0 [4716] Simulating particle 1 [4716] Simulating particle 2 [4716] Simulating particle 3 [4716] Simulating particle 4 [4716] Simulating particle 5 [4716] Simulating particle 6 [4716] Simulating particle 7 [4716] Simulating particle 8 [4716] Simulating particle 9 [4716] Simulating particle 10 [4716] Simulating particle 11 [4716] Simulating particle 12 [4716] Simulating particle 13 [4716] Simulating particle 14 [4716] Simulating particle 15 [4716] Simulating particle 16 [4716] Simulating particle 17 [4716] Simulating particle 18 [4716] Simulating particle 19 [4716] Simulating particle 20 [4716] Simulating particle 21 [4716] Simulating particle 22 [4716] Simulating particle 23 [4716] Simulating particle 24 [4716] Simulating particle 25 [4716] Simulating particle 26 [4716] Simulating particle 27 [4716] Simulating particle 28 [4716] Simulating particle 29 [stdout:3] [4636] Simulating particle 0 [4636] Simulating particle 1 [4636] Simulating particle 2 [4636] Simulating particle 3 [4636] Simulating particle 4 [4636] Simulating particle 5 [4636] Simulating particle 6 [4636] Simulating particle 7 [4636] Simulating particle 8 [4636] Simulating particle 9 [4636] Simulating particle 10 [4636] Simulating particle 11 [4636] Simulating particle 12 [4636] Simulating particle 13 [4636] Simulating particle 14 [4636] Simulating particle 15 [4636] Simulating particle 16 [4636] Simulating particle 17 [4636] Simulating particle 18 [4636] Simulating particle 19 [4636] Simulating particle 20 [4636] Simulating particle 21 [4636] Simulating particle 22 [4636] Simulating particle 23 [4636] Simulating particle 24 [4636] Simulating particle 25 [4636] Simulating particle 26 [4636] Simulating particle 27 [4636] Simulating particle 28 [4636] Simulating particle 29 [stdout:4] [4352] Simulating particle 0 [4352] Simulating particle 1 [4352] Simulating particle 2 [4352] Simulating particle 3 [4352] Simulating particle 4 [4352] Simulating particle 5 [4352] Simulating particle 6 [4352] Simulating particle 7 [4352] Simulating particle 8 [4352] Simulating particle 9 [4352] Simulating particle 10 [4352] Simulating particle 11 [4352] Simulating particle 12 [4352] Simulating particle 13 [4352] Simulating particle 14 [4352] Simulating particle 15 [4352] Simulating particle 16 [4352] Simulating particle 17 [4352] Simulating particle 18 [4352] Simulating particle 19 [4352] Simulating particle 20 [4352] Simulating particle 21 [4352] Simulating particle 22 [4352] Simulating particle 23 [4352] Simulating particle 24 [4352] Simulating particle 25 [4352] Simulating particle 26 [4352] Simulating particle 27 [4352] Simulating particle 28 [4352] Simulating particle 29 [stdout:5] [5008] Simulating particle 0 [5008] Simulating particle 1 [5008] Simulating particle 2 [5008] Simulating particle 3 [5008] Simulating particle 4 [5008] Simulating particle 5 [5008] Simulating particle 6 [5008] Simulating particle 7 [5008] Simulating particle 8 [5008] Simulating particle 9 [5008] Simulating particle 10 [5008] Simulating particle 11 [5008] Simulating particle 12 [5008] Simulating particle 13 [5008] Simulating particle 14 [5008] Simulating particle 15 [5008] Simulating particle 16 [5008] Simulating particle 17 [5008] Simulating particle 18 [5008] Simulating particle 19 [5008] Simulating particle 20 [5008] Simulating particle 21 [5008] Simulating particle 22 [5008] Simulating particle 23 [5008] Simulating particle 24 [5008] Simulating particle 25 [5008] Simulating particle 26 [5008] Simulating particle 27 [5008] Simulating particle 28 [5008] Simulating particle 29 [stdout:6] [1944] Simulating particle 0 [1944] Simulating particle 1 [1944] Simulating particle 2 [1944] Simulating particle 3 [1944] Simulating particle 4 [1944] Simulating particle 5 [1944] Simulating particle 6 [1944] Simulating particle 7 [1944] Simulating particle 8 [1944] Simulating particle 9 [1944] Simulating particle 10 [1944] Simulating particle 11 [1944] Simulating particle 12 [1944] Simulating particle 13 [1944] Simulating particle 14 [1944] Simulating particle 15 [1944] Simulating particle 16 [1944] Simulating particle 17 [1944] Simulating particle 18 [1944] Simulating particle 19 [1944] Simulating particle 20 [1944] Simulating particle 21 [1944] Simulating particle 22 [1944] Simulating particle 23 [1944] Simulating particle 24 [1944] Simulating particle 25 [1944] Simulating particle 26 [1944] Simulating particle 27 [1944] Simulating particle 28 [1944] Simulating particle 29 [stdout:7] [3744] Simulating particle 0 [3744] Simulating particle 1 [3744] Simulating particle 2 [3744] Simulating particle 3 [3744] Simulating particle 4 [3744] Simulating particle 5 [3744] Simulating particle 6 [3744] Simulating particle 7 [3744] Simulating particle 8 [3744] Simulating particle 9 [3744] Simulating particle 10 [3744] Simulating particle 11 [3744] Simulating particle 12 [3744] Simulating particle 13 [3744] Simulating particle 14 [3744] Simulating particle 15 [3744] Simulating particle 16 [3744] Simulating particle 17 [3744] Simulating particle 18 [3744] Simulating particle 19 [3744] Simulating particle 20 [3744] Simulating particle 21 [3744] Simulating particle 22 [3744] Simulating particle 23 [3744] Simulating particle 24 [3744] Simulating particle 25 [3744] Simulating particle 26 [3744] Simulating particle 27 [3744] Simulating particle 28 [3744] Simulating particle 29
#!beep #Make some noise
Save the simulation to a file (so it can be reloaded):
%%px
S.dump(dir_=SIM_DATA_DIR)
[stdout:0] Saved to C:/Data/Antonio/data/sim/brownian/objects//bromo_sim_D1.2e-11_30P_64pM_step0.5us_t_max10.0s_ID0-6.pickle [stdout:1] Saved to C:/Data/Antonio/data/sim/brownian/objects//bromo_sim_D1.2e-11_30P_64pM_step0.5us_t_max10.0s_ID1-6.pickle [stdout:2] Saved to C:/Data/Antonio/data/sim/brownian/objects//bromo_sim_D1.2e-11_30P_64pM_step0.5us_t_max10.0s_ID2-6.pickle [stdout:3] Saved to C:/Data/Antonio/data/sim/brownian/objects//bromo_sim_D1.2e-11_30P_64pM_step0.5us_t_max10.0s_ID3-6.pickle [stdout:4] Saved to C:/Data/Antonio/data/sim/brownian/objects//bromo_sim_D1.2e-11_30P_64pM_step0.5us_t_max10.0s_ID4-6.pickle [stdout:5] Saved to C:/Data/Antonio/data/sim/brownian/objects//bromo_sim_D1.2e-11_30P_64pM_step0.5us_t_max10.0s_ID5-6.pickle [stdout:6] Saved to C:/Data/Antonio/data/sim/brownian/objects//bromo_sim_D1.2e-11_30P_64pM_step0.5us_t_max10.0s_ID6-6.pickle [stdout:7] Saved to C:/Data/Antonio/data/sim/brownian/objects//bromo_sim_D1.2e-11_30P_64pM_step0.5us_t_max10.0s_ID7-6.pickle
Load the simulation and check that is the same as the one in memory:
%%px
S2 = load_sim_id(ID=0, glob_str="*t_max10.0s*", dir_=SIM_DATA_DIR)
[stdout:4] Loaded: C:/Data/Antonio/data/sim/brownian/objects\bromo_sim_D1.2e-11_30P_64pM_step0.5us_t_max10.0s_ID4-0.pickle [stdout:5] Loaded: C:/Data/Antonio/data/sim/brownian/objects\bromo_sim_D1.2e-11_30P_64pM_step0.5us_t_max10.0s_ID5-0.pickle [stdout:6] Loaded: C:/Data/Antonio/data/sim/brownian/objects\bromo_sim_D1.2e-11_30P_64pM_step0.5us_t_max10.0s_ID6-0.pickle [stdout:7] Loaded: C:/Data/Antonio/data/sim/brownian/objects\bromo_sim_D1.2e-11_30P_64pM_step0.5us_t_max10.0s_ID7-0.pickle
%%px
(S.em == S2.em).all() # Return `True` if arrays equal
Out[0:35]: True
Out[1:35]: True
Out[2:35]: True
Out[3:35]: True
Out[4:35]: True
Out[5:35]: True
Out[6:35]: True
Out[7:35]: True
%px del S2
The following assumes you perfomed at least two simulations (with ID 0 and 1) on the remote engines
This is a quick example. For a better description see the next notebook: 2. Generate timestamps - Parallel
Here we use the cluster to load the emission traces from file for two simulation set (ID=0 and ID=1).
Then we generate the timestamps (ph_times
) for the donor (D) and acceptor (D)
channels.
IDs = [0,1]
for ID in IDs:
dview['ID'] = ID
%px S = load_sim_id(ID=ID, glob_str="*t_max10.0s*", dir_=SIM_DATA_DIR)
[stdout:4] Loaded: C:/Data/Antonio/data/sim/brownian/objects\bromo_sim_D1.2e-11_30P_64pM_step0.5us_t_max10.0s_ID4-0.pickle [stdout:5] Loaded: C:/Data/Antonio/data/sim/brownian/objects\bromo_sim_D1.2e-11_30P_64pM_step0.5us_t_max10.0s_ID5-0.pickle [stdout:6] Loaded: C:/Data/Antonio/data/sim/brownian/objects\bromo_sim_D1.2e-11_30P_64pM_step0.5us_t_max10.0s_ID6-0.pickle [stdout:7] Loaded: C:/Data/Antonio/data/sim/brownian/objects\bromo_sim_D1.2e-11_30P_64pM_step0.5us_t_max10.0s_ID7-0.pickle [stdout:4] Loaded: C:/Data/Antonio/data/sim/brownian/objects\bromo_sim_D1.2e-11_30P_64pM_step0.5us_t_max10.0s_ID4-1.pickle [stdout:5] Loaded: C:/Data/Antonio/data/sim/brownian/objects\bromo_sim_D1.2e-11_30P_64pM_step0.5us_t_max10.0s_ID5-1.pickle [stdout:6] Loaded: C:/Data/Antonio/data/sim/brownian/objects\bromo_sim_D1.2e-11_30P_64pM_step0.5us_t_max10.0s_ID6-1.pickle [stdout:7] Loaded: C:/Data/Antonio/data/sim/brownian/objects\bromo_sim_D1.2e-11_30P_64pM_step0.5us_t_max10.0s_ID7-1.pickle
ph_times_d, t_tot, sim_name = parallel_gen_timestamps(dview,
max_em_rate=1e6, bg_rate=2e3)
ph_times_a, t_tot, sim_name = parallel_gen_timestamps(dview,
max_em_rate=5e5, bg_rate=4e3)
Now we merge the two arrays ph_times_d
and ph_times_a
in a single array ph_times
and use a boolean mask (array of booleans) a_em
to signal if each timestamp
is from the Acceptor (True
) or from the Donor (False
).
ph_times, a_em = merge_DA_ph_times(ph_times_d, ph_times_a)
Just test that the new arrays are consistent with the old ones:
(ph_times_d == ph_times[-a_em]).all()
True
(ph_times_a == ph_times[a_em]).all()
True
import matplotlib.pyplot as plt
#%matplotlib qt
Plot the full array of timestamps (the merged timestamps from the cluster):
plt.plot(ph_times)
[<matplotlib.lines.Line2D at 0x55eee10>]
plt.hist(ph_times, bins=arange(0,4,1e-3), histtype='step');
Plot the emission for the simulation in one engine (no need to transfer the array):
%%px --target 5
%matplotlib inline
plot_emission(S, ms=1)
plt.xlim(0,1000)
Out[5:1317]: (0, 1000)
%%px
plt.close('all')
Plot the emission for the simulation loaded in ALL the engines:
%%px
%matplotlib inline
plot_emission(S, ms=1)
#plt.xlim(25,35)
[output:4]
[output:5]
[output:6]
[output:7]
*NOTE:* Here I'm just keeping track of some useful command
%px %cd 'C:\Data\Antonio\data\sim\brownian\traces_em'
[stdout:5] C:\Data\Antonio\data\sim\brownian\traces_em [stdout:6] C:\Data\Antonio\data\sim\brownian\traces_em [stdout:7] C:\Data\Antonio\data\sim\brownian\traces_em [stdout:8] C:\Data\Antonio\data\sim\brownian\traces_em
%px import os
%px from glob import glob
%%px --target 5
# Execute this only on ONE engine
for f in glob('*'):
print f
#os.remove(f)
from IPython.core.display import HTML
def css_styling():
styles = open("./styles/custom2.css", "r").read()
return HTML(styles)
css_styling()