from dmpipe.pipeline import Pipeline
/nfs/slac/kipac/fs1/u/dmcat/software/anaconda2/envs/dmcat-gitdev/lib/python2.7/site-packages/matplotlib/__init__.py:1350: UserWarning: This call to matplotlib.use() has no effect because the backend has already been chosen; matplotlib.use() must be called *before* pylab, matplotlib.pyplot, or matplotlib.backends is imported for the first time. warnings.warn(_use_error_msg)
Using /nfs/slac/kipac/fs1/u/dmcat/software/anaconda2/envs/dmcat-gitdev/lib/python2.7/site-packages/dmsky-0.2.1-py2.7.egg/dmsky/data/targets for TargetLibrary Using /gpfs/slac/kipac/fs1/u/dmcat/workdir/sandbox/test_dmpipe/dmsky_data/targets for TargetLibrary
First we build a pipeline object to analyze the dSphs, and point it at the config file for the dSphs analysis.
pipe = Pipeline(linkname = 'dSphs')
configfile = 'config/master_dSphs.yaml'
We have to 'preconfigure' the pipeline, because we need to build up the list of targets so that we correctly set up the later stages of the pipeline.
pipe.preconfigure(configfile)
return code 0
We then tell the pipeline to update the arguments for all of the steps that comprise it.
pipe.update_args(dict(config=configfile))
Now we look around a bit, we can drill down into the links that make up the pipeline. In this case we have a pipeline that consists of a couple of preparation jobs, then 5 sub-pipelines ('data', 'sim_xxx', 'random'). The sub-pipeline analyze the real data, and some different simulation cases. The 'random' sub-pipeline analyzes random sky directions.
At this point the pipeline is fully configured, let's look around.
pipe.linknames
['prepare-targets', 'spec-table', 'data', 'sim_100GeV_bb_3e26', 'sim_null', 'sim_100GeV_bb_1e25', 'random', 'final-plots']
pipe['data']
<dmpipe.pipeline.PipelineData at 0x7f8347d897d0>
pipe['data'].linknames
['analyze-roi', 'analyze-sed', 'convert-castro', 'stack-likelihood', 'plot-castro-sg', 'plot-dm-sg', 'plot-limits-sg', 'plot-stacked-dm-sg', 'plot-stacked-limits-sg']
We can print the status for the various links. Using recurse=True will also drill down to the status of the links in the sub pipelines.
pipe.print_status()
Linkname : Link Status : Jobs Status prepare-targets : Done : Done spec-table : Ready : Pending data : Ready : Pending sim_100GeV_bb_3e26 : Ready : Pending sim_null : Ready : Pending sim_100GeV_bb_1e25 : Ready : Pending random : Ready : Pending final-plots : Ready : 8/ 0/ 0/ 0/ 0/ 8
pipe.print_status(recurse=True)
Linkname : Link Status : Jobs Status prepare-targets : Done : Done spec-table : Ready : Pending data : Ready : Pending ---------- data ----------- Linkname : Link Status : Jobs Status analyze-roi : Ready : 2/ 0/ 0/ 0/ 0/ 2 analyze-sed : Ready : 2/ 0/ 0/ 0/ 0/ 2 convert-castro : Ready : 4/ 0/ 0/ 0/ 0/ 4 stack-likelihood : Ready : 2/ 0/ 0/ 0/ 0/ 2 plot-castro-sg : Ready : 2/ 0/ 0/ 0/ 0/ 2 plot-dm-sg : Ready : 8/ 0/ 0/ 0/ 0/ 8 plot-limits-sg : Ready : 8/ 0/ 0/ 0/ 0/ 8 plot-stacked-dm-sg : Ready : 4/ 0/ 0/ 0/ 0/ 4 plot-stacked-limits-sg : Ready : 4/ 0/ 0/ 0/ 0/ 4 ------------------------------------------------ sim_100GeV_bb_3e26 : Ready : Pending ---------- sim_100GeV_bb_3e26 ----------- Linkname : Link Status : Jobs Status copy-base-roi : Ready : 2/ 0/ 0/ 0/ 0/ 2 simulate-roi : Ready : 2/ 0/ 0/ 0/ 0/ 2 convert-castro : Ready : 4/ 0/ 0/ 0/ 0/ 4 stack-likelihood : Ready : 2/ 0/ 0/ 0/ 0/ 2 collect-sed : Ready : 2/ 0/ 0/ 0/ 0/ 2 collect-limits : Ready : 4/ 0/ 0/ 0/ 0/ 4 collect-stacked-limits : Ready : 2/ 0/ 0/ 0/ 0/ 2 plot-stacked-dm : Ready : 80/ 0/ 0/ 0/ 0/ 80 plot-stacked-limits : Ready : 80/ 0/ 0/ 0/ 0/ 80 plot-control-limits : Ready : 4/ 0/ 0/ 0/ 0/ 4 ------------------------------------------------ sim_null : Ready : Pending ---------- sim_null ----------- Linkname : Link Status : Jobs Status copy-base-roi : Ready : 2/ 0/ 0/ 0/ 0/ 2 simulate-roi : Ready : 2/ 0/ 0/ 0/ 0/ 2 convert-castro : Ready : 4/ 0/ 0/ 0/ 0/ 4 stack-likelihood : Ready : 2/ 0/ 0/ 0/ 0/ 2 collect-sed : Ready : 2/ 0/ 0/ 0/ 0/ 2 collect-limits : Ready : 4/ 0/ 0/ 0/ 0/ 4 collect-stacked-limits : Ready : 2/ 0/ 0/ 0/ 0/ 2 plot-stacked-dm : Ready : 80/ 0/ 0/ 0/ 0/ 80 plot-stacked-limits : Ready : 80/ 0/ 0/ 0/ 0/ 80 plot-control-limits : Ready : 4/ 0/ 0/ 0/ 0/ 4 ------------------------------------------------ sim_100GeV_bb_1e25 : Ready : Pending ---------- sim_100GeV_bb_1e25 ----------- Linkname : Link Status : Jobs Status copy-base-roi : Ready : 2/ 0/ 0/ 0/ 0/ 2 simulate-roi : Ready : 2/ 0/ 0/ 0/ 0/ 2 convert-castro : Ready : 4/ 0/ 0/ 0/ 0/ 4 stack-likelihood : Ready : 2/ 0/ 0/ 0/ 0/ 2 collect-sed : Ready : 2/ 0/ 0/ 0/ 0/ 2 collect-limits : Ready : 4/ 0/ 0/ 0/ 0/ 4 collect-stacked-limits : Ready : 2/ 0/ 0/ 0/ 0/ 2 plot-stacked-dm : Ready : 80/ 0/ 0/ 0/ 0/ 80 plot-stacked-limits : Ready : 80/ 0/ 0/ 0/ 0/ 80 plot-control-limits : Ready : 4/ 0/ 0/ 0/ 0/ 4 ------------------------------------------------ random : Ready : Pending ---------- random ----------- Linkname : Link Status : Jobs Status copy-base-roi : Ready : 2/ 0/ 0/ 0/ 0/ 2 random-dir-gen : Ready : 2/ 0/ 0/ 0/ 0/ 2 analyze-sed : Ready : 2/ 0/ 0/ 0/ 0/ 2 convert-castro : Ready : 4/ 0/ 0/ 0/ 0/ 4 stack-likelihood : Ready : 2/ 0/ 0/ 0/ 0/ 2 collect-sed : Ready : 2/ 0/ 0/ 0/ 0/ 2 collect-limits : Ready : 4/ 0/ 0/ 0/ 0/ 4 collect-stacked-limits : Ready : 2/ 0/ 0/ 0/ 0/ 2 plot-stacked-dm : Ready : 80/ 0/ 0/ 0/ 0/ 80 plot-stacked-limits : Ready : 80/ 0/ 0/ 0/ 0/ 80 ------------------------------------------------ final-plots : Ready : 8/ 0/ 0/ 0/ 0/ 8
We can access a particular Link in the pipeline.
We can ask:
what the default options for the link are ('_options')
what the current set of options are ('args')
pipe['data']['analyze-roi']
<fermipy.jobs.target_analysis.AnalyzeROI_SG at 0x7f8348281210>
pipe['data']['analyze-roi']._options
{'action': ('run', 'Action to perform', str), 'check_status_once': (False, 'Check status only once before proceeding', bool), 'config': (None, 'Path to fermipy config file.', str), 'dry_run': (False, 'Print commands, but do not execute them', bool), 'job_check_sleep': (300, 'Sleep time between checking on job status (s)', int), 'make_plots': (False, 'Make plots', bool), 'print_update': (False, 'Print summary of job status', bool), 'roi_baseline': ('fit_baseline', 'Key for roi baseline file.', str), 'targetlist': (None, 'Path to the target list.', str), 'ttype': (None, 'Type of target being analyzed.', str)}
pipe['data']['analyze-roi'].args
{'action': 'run', 'check_status_once': False, 'config': None, 'dry_run': False, 'job_check_sleep': 300, 'make_plots': False, 'print_update': False, 'roi_baseline': 'fit_baseline', 'targetlist': None, 'ttype': 'dSphs'}
We can ask what jobs get run by this link. In this paritcular case the link runs 1 job that in turn dispatches several other jobs to the batch farm.
pipe['data']['analyze-roi'].jobs
OrderedDict([('__top__@dSphs.data.analyze-roi', jobname : dSphs.data.analyze-roi jobkey : __top__ appname : fermipy-analyze-roi-sg logfile : logs/dSphs.data.analyze-roi.log status : Ready)])
Here we are talking to the link that represents any one of the disptached jobs. The command_template() function tells us how we would run this job from the UNIX command line.
pipe['data']['analyze-roi'].scatter_link
<fermipy.jobs.target_analysis.AnalyzeROI at 0x7f83933d0f50>
pipe['data']['analyze-roi'].scatter_link.command_template()
'fermipy-analyze-roi --roi_baseline {roi_baseline} --config {config}'
Here are ask what jobs will be dispatched. Note that there are two jobs with slightly different names.
pipe['data']['analyze-roi'].scatter_link.jobs
OrderedDict([('draco@dSphs.data.analyze-roi', jobname : dSphs.data.analyze-roi jobkey : draco appname : fermipy-analyze-roi logfile : /nfs/slac/kipac/fs1/u/dmcat/workdir/sandbox/dmpipe_example/dSphs/draco/analyze-roi_draco.log status : Ready), ('segue_1@dSphs.data.analyze-roi', jobname : dSphs.data.analyze-roi jobkey : segue_1 appname : fermipy-analyze-roi logfile : /nfs/slac/kipac/fs1/u/dmcat/workdir/sandbox/dmpipe_example/dSphs/segue_1/analyze-roi_segue_1.log status : Ready)])
Here are are asking for information about the first of those two jobs. In particular, the specific options used for this instance of the job. You can merge the job_config with the command_template to get the exact syntax for the instance of this command.
pipe['data']['analyze-roi'].scatter_link.jobs['draco@dSphs.data.analyze-roi']
jobname : dSphs.data.analyze-roi jobkey : draco appname : fermipy-analyze-roi logfile : /nfs/slac/kipac/fs1/u/dmcat/workdir/sandbox/dmpipe_example/dSphs/draco/analyze-roi_draco.log status : Ready
pipe['data']['analyze-roi'].scatter_link.jobs['draco@dSphs.data.analyze-roi'].job_config
{'config': './dSphs/draco/config.yaml', 'logfile': '/nfs/slac/kipac/fs1/u/dmcat/workdir/sandbox/dmpipe_example/dSphs/draco/analyze-roi_draco.log', 'make_plots': False, 'roi_baseline': 'fit_baseline'}
Here we run a single link.
We use 'run_with_log()' so that the log file will be in the correct place and the pipeline will know the job
has completed.
pipe['spec-table'].run_with_log()
return code 0
pipe.print_status()
Linkname : Link Status : Jobs Status prepare-targets : Done : Done spec-table : Done : Done data : Ready : Pending sim_100GeV_bb_3e26 : Ready : Pending sim_null : Ready : Pending sim_100GeV_bb_1e25 : Ready : Pending random : Ready : Pending final-plots : Ready : 8/ 0/ 0/ 0/ 0/ 8
Here we run the 'data' sub-pipeline.
pipe['data'].run(resubmit_failed=True)
Running link dSphs.data.analyze-roi Submitting jobs (2): ! Checking status (300s): ...! Done Running link dSphs.data.analyze-sed Submitting jobs (2): ! Checking status (300s): .! Done Running link dSphs.data.convert-castro Submitting jobs (4): ! Checking status (120s): .! Done Running link dSphs.data.stack-likelihood Submitting jobs (2): ! Checking status (60s): .! Done Running link dSphs.data.plot-castro-sg Submitting jobs (2): ! Checking status (60s): .! Done Running link dSphs.data.plot-dm-sg Submitting jobs (8): ! Checking status (60s): .! Done Running link dSphs.data.plot-limits-sg Submitting jobs (8): ! Checking status (60s): .! Done Running link dSphs.data.plot-stacked-dm-sg Submitting jobs (4): ! Checking status (60s): .! Done Running link dSphs.data.plot-stacked-limits-sg Submitting jobs (4): ! Checking status (60s): .! Done Chain status 5 Timestamp: 1527130057 Successfully completed
pipe.print_status(recurse=True)
Linkname : Link Status : Jobs Status prepare-targets : Done : Done spec-table : Done : Done data : Done : Done ---------- data ----------- Linkname : Link Status : Jobs Status analyze-roi : Done : 0/ 0/ 0/ 2/ 0/ 2 analyze-sed : Done : 0/ 0/ 0/ 2/ 0/ 2 convert-castro : Done : 0/ 0/ 0/ 4/ 0/ 4 stack-likelihood : Done : 0/ 0/ 0/ 2/ 0/ 2 plot-castro-sg : Done : 0/ 0/ 0/ 2/ 0/ 2 plot-dm-sg : Done : 0/ 0/ 0/ 8/ 0/ 8 plot-limits-sg : Done : 0/ 0/ 0/ 8/ 0/ 8 plot-stacked-dm-sg : Done : 0/ 0/ 0/ 4/ 0/ 4 plot-stacked-limits-sg : Done : 0/ 0/ 0/ 4/ 0/ 4 ------------------------------------------------ sim_100GeV_bb_3e26 : Ready : Pending ---------- sim_100GeV_bb_3e26 ----------- Linkname : Link Status : Jobs Status copy-base-roi : Ready : 2/ 0/ 0/ 0/ 0/ 2 simulate-roi : Ready : 2/ 0/ 0/ 0/ 0/ 2 convert-castro : Ready : 4/ 0/ 0/ 0/ 0/ 4 stack-likelihood : Ready : 2/ 0/ 0/ 0/ 0/ 2 collect-sed : Ready : 2/ 0/ 0/ 0/ 0/ 2 collect-limits : Ready : 4/ 0/ 0/ 0/ 0/ 4 collect-stacked-limits : Ready : 2/ 0/ 0/ 0/ 0/ 2 plot-stacked-dm : Ready : 80/ 0/ 0/ 0/ 0/ 80 plot-stacked-limits : Ready : 80/ 0/ 0/ 0/ 0/ 80 plot-control-limits : Ready : 4/ 0/ 0/ 0/ 0/ 4 ------------------------------------------------ sim_null : Ready : Pending ---------- sim_null ----------- Linkname : Link Status : Jobs Status copy-base-roi : Ready : 2/ 0/ 0/ 0/ 0/ 2 simulate-roi : Ready : 2/ 0/ 0/ 0/ 0/ 2 convert-castro : Ready : 4/ 0/ 0/ 0/ 0/ 4 stack-likelihood : Ready : 2/ 0/ 0/ 0/ 0/ 2 collect-sed : Ready : 2/ 0/ 0/ 0/ 0/ 2 collect-limits : Ready : 4/ 0/ 0/ 0/ 0/ 4 collect-stacked-limits : Ready : 2/ 0/ 0/ 0/ 0/ 2 plot-stacked-dm : Ready : 80/ 0/ 0/ 0/ 0/ 80 plot-stacked-limits : Ready : 80/ 0/ 0/ 0/ 0/ 80 plot-control-limits : Ready : 4/ 0/ 0/ 0/ 0/ 4 ------------------------------------------------ sim_100GeV_bb_1e25 : Ready : Pending ---------- sim_100GeV_bb_1e25 ----------- Linkname : Link Status : Jobs Status copy-base-roi : Ready : 2/ 0/ 0/ 0/ 0/ 2 simulate-roi : Ready : 2/ 0/ 0/ 0/ 0/ 2 convert-castro : Ready : 4/ 0/ 0/ 0/ 0/ 4 stack-likelihood : Ready : 2/ 0/ 0/ 0/ 0/ 2 collect-sed : Ready : 2/ 0/ 0/ 0/ 0/ 2 collect-limits : Ready : 4/ 0/ 0/ 0/ 0/ 4 collect-stacked-limits : Ready : 2/ 0/ 0/ 0/ 0/ 2 plot-stacked-dm : Ready : 80/ 0/ 0/ 0/ 0/ 80 plot-stacked-limits : Ready : 80/ 0/ 0/ 0/ 0/ 80 plot-control-limits : Ready : 4/ 0/ 0/ 0/ 0/ 4 ------------------------------------------------ random : Ready : Pending ---------- random ----------- Linkname : Link Status : Jobs Status copy-base-roi : Ready : 2/ 0/ 0/ 0/ 0/ 2 random-dir-gen : Ready : 2/ 0/ 0/ 0/ 0/ 2 analyze-sed : Ready : 2/ 0/ 0/ 0/ 0/ 2 convert-castro : Ready : 4/ 0/ 0/ 0/ 0/ 4 stack-likelihood : Ready : 2/ 0/ 0/ 0/ 0/ 2 collect-sed : Ready : 2/ 0/ 0/ 0/ 0/ 2 collect-limits : Ready : 4/ 0/ 0/ 0/ 0/ 4 collect-stacked-limits : Ready : 2/ 0/ 0/ 0/ 0/ 2 plot-stacked-dm : Ready : 80/ 0/ 0/ 0/ 0/ 80 plot-stacked-limits : Ready : 80/ 0/ 0/ 0/ 0/ 80 ------------------------------------------------ final-plots : Ready : 8/ 0/ 0/ 0/ 0/ 8
pipe['data']['convert-castro'].scatter_link.jobs['draco:ack2016_point:lgauss@dSphs.data.convert-castro'].job_config
{'clobber': False, 'j_value_file': './dSphs/draco/j_val_ack2016.yaml', 'jprior': 'lgauss', 'limitfile': './dSphs/draco/dmlimits_ack2016_point_lgauss.fits', 'logfile': '/nfs/slac/kipac/fs1/u/dmcat/workdir/sandbox/dmpipe_example/dSphs/draco/dmlike_ack2016_point_lgauss.log', 'nsims': -1, 'outfile': './dSphs/draco/dmlike_ack2016_point_lgauss.fits', 'sed_file': './dSphs/draco/sed_ack2016_point.fits', 'seed': -1, 'specfile': './dm_spectra_dSphs.fits'}
pipe.run(resubmit_failed=True)
Skipping done link dSphs.prepare-targets Skipping done link dSphs.spec-table Running link dSphs.data.data Skipping done link dSphs.data.analyze-roi Skipping done link dSphs.data.analyze-sed Skipping done link dSphs.data.convert-castro Skipping done link dSphs.data.stack-likelihood Skipping done link dSphs.data.plot-castro-sg Skipping done link dSphs.data.plot-dm-sg Skipping done link dSphs.data.plot-limits-sg Skipping done link dSphs.data.plot-stacked-dm-sg Skipping done link dSphs.data.plot-stacked-limits-sg Chain status 5 Running link dSphs.sim_100GeV_bb_3e26.sim_100GeV_bb_3e26 Running link dSphs.sim_100GeV_bb_3e26.copy-base-roi Submitting jobs (2): ! Checking status (60s): .! Done Running link dSphs.sim_100GeV_bb_3e26.simulate-roi Submitting jobs (2): ! Checking status (300s): ......! Done Running link dSphs.sim_100GeV_bb_3e26.convert-castro Submitting jobs (4): ! Checking status (120s): ...! Done Running link dSphs.sim_100GeV_bb_3e26.stack-likelihood Submitting jobs (2): ! Checking status (60s): ..! Done Running link dSphs.sim_100GeV_bb_3e26.collect-sed Submitting jobs (2): ! Checking status (60s): .! Done Running link dSphs.sim_100GeV_bb_3e26.collect-limits Submitting jobs (4): ! Checking status (60s): ...! Partially Failed Stoping chain execution at failed link dSphs.sim_100GeV_bb_3e26.collect-limits Chain status 7 Stoping chain execution at failed link dSphs.sim_100GeV_bb_3e26.sim_100GeV_bb_3e26 Chain status 7 Timestamp: 1527132646 Exited with exit code -1