The Bootstrap Analysis of Stable Clusters pipeline

This tutorial shows how to run a Bootstrap Analysis of Stable Clusters (BASC) pipeline using NIAK.

  • More documentation: This tutorial only shows a limited subset of available options. See the documentation of the pipeline for a more comprehensive list of options.
  • .m script: The script of the tutorial can be downloaded here and the notebook is available here.
  • Pre-requisites: before running this tutorial, you need to complete the fMRI preprocessing tutorial.
  • Time for completion: this tutorial will take approximetly 15 minutes to run.

Prepare input files

fMRI

We will re-use the fMRI data that was generated during the fMRI preprocessing tutorial. The following instructions are going to "grab" these data. A few options are available to select subjects based on the level of motion, number of available volumes or run name. It is also possible to exclude subjects explicitely, e.g. if they failed visual quality control assessment.

In [1]:
% Where to find data
clear
path_data = [pwd filesep];
path_preprocess = [path_data 'fmri_preprocess'];

% That option tells the grabber to prepare files for BASC
opt_g.type_files = 'rest'; 

% Minimum number of volumes per run - typically aim at 2 mns
% Shorther here because the test dataset is really small
opt_g.min_nb_vol = 20;

% Grab the "motor" run
opt_g.filter.run = {'motor'}; % Just grab the "motor" runs

% if uncommented, this option would exclude subject1
% opt_g.exclude_subject = {'subject1'};

% Grab the data
files_in = niak_grab_fmri_preprocess(path_preprocess,opt_g)
files_in =

  scalar structure containing the fields:

    data =

      scalar structure containing the fields:

        subject1 =

          scalar structure containing the fields:

            session1: 1x1 scalar struct

        subject2 =

          scalar structure containing the fields:

            session1: 1x1 scalar struct


    mask = /sandbox/home/git/niak_tutorials/fmri_preprocess/quality_control/group_coregistration/func_mask_group_stereonl.nii.gz
    areas = /sandbox/home/git/niak_tutorials/fmri_preprocess/anat/template_aal.nii.gz

Set the options of the pipeline

Now we set up where to store the results:

In [2]:
% Where to store the results
opt.folder_out = [path_data 'basc_test_niak'];

This next parameter sets the size at which the region growing process stops (in mm$^3$). This does indirectly set the number of parcels covering the gray matter. A threshold of 1000 mm$^3$ will give about 1000 regions on the grey matter. Here we are going to reduce the dimension even more to speed up the pipeline.

In [3]:
% the size of the regions, when they stop growing. 
opt.region_growing.thre_size = 2000;

The following parameter sets the values that will be explored regarding the number of clusters. A more exhaustive search, up to 500 clusters, could for example use an irregular grid like [10:10:100 120:20:200 240:40:500]. Also note that for each number of clusters, a number of solutions around those values, with different numbers being used at the individual and group levels, as well as the final consensus group partitions.

In [4]:
% Search for stable clusters in the range 10 to 30
opt.grid_scales = [10:10:30]';

The following parameter is used to generate stability maps, and consensus partitions. Although stability is assessed over a wide range of parameters, those maps are only generated for select numbers in order to save disk space. For each set of results, three (integer) parameters actually need to be provided: the number of clusters at the individual level, at the group level and at the final consensus level. Each row will define a new set of results. We could for example use opt.scales_maps = [10 10 10; 20 20 20]; to generate maps using the same number of clusters (5 and 10) at all three levels. For now we will leave that empty, which means we will not be generating any map. A first run of the pipeline is going to give us insights in the stable cluster solution, and provide data-driven suggestions about what number(s) of clusters to use.

In [5]:
% Scale parameters to generate stability maps and consensus clusters
opt.scales_maps = [];

The following parameters control the number of bootstrap samples used to assess the stability of the clustering both at the individual level, and the group level. Note that with 100 samples, the confidence interval on the stability measures is $\pm 0.1$, while with 500 samples it reaches $\pm 0.05$. Here we will use only a few (20 samples) to speed-up the pipeline.

In [6]:
% Number of bootstrap samples at the individual level.
opt.stability_tseries.nb_samps = 20;
% Number of bootstrap samples at the group level. 
opt.stability_group.nb_samps = 20;

Also note that there is a minimal number of subjects for the group stability analysis to run. Because this demo has only two subjects, we will need to lower this parameter:

In [7]:
opt.stability_group.min_subject = 2;

The final set of flags tell the pipeline which level of the pipeline to run. BASC can be used at the individual level, for each subject independently, or at the group level, to find a parcellation that is the consensus of those generated across all subjects and bootstrap replications of individual fMRI time series. In addition, BASC can generate so-called "mixed parcellation", which individual parcellations generated using the group parcellation as an initialization. It is possible to turn on/off the generation of consensus parcellations and stability map for each level of the pipeline (individual, group or mixed):

In [8]:
% Generate maps/time series at the individual level
opt.flag_ind = false;   
% Generate maps/time series at the mixed level (group-level networks mixed with individual stability matrices).
opt.flag_mixed = false; 
% Generate maps/time series at the group level
opt.flag_group = true;

Running the pipeline

Now it is time to run the pipeline. As explained above, we will not at this stage generate any map or parcellation, but we will use the outputs to select the scales that we will explore in more details. The instruction to run the pipeline is:

In [9]:
niak_pipeline_stability_rest(files_in,opt);
Logs will be stored in /sandbox/home/git/niak_tutorials/basc_test_niak/logs/
Generating dependencies ...
   Percentage completed :  0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100- 0.39 sec
Setting up the to-do list ...
   I found 133 job(s) to do.
I could not find any log file. This pipeline has not been started (yet?). Press CTRL-C to cancel.
Deamon started on 30-Apr-2017 01:22:29
30-Apr-2017 01:22:29 Starting the pipeline manager...
30-Apr-2017 01:22:30 Starting the garbage collector...
30-Apr-2017 01:22:30 Starting worker number 1...
30-Apr-2017 01:22:30 Starting worker number 2...
30-Apr-2017 01:22:30 Starting worker number 3...
30-Apr-2017 01:22:30 Starting worker number 4...

Pipeline started on 30-Apr-2017 01:22:32
user: , host: 3f86dc5d68dc, system: unix
****************************************
30-Apr-2017 01:22:34 mask_areas                            submitted  (1 run | 0 fail | 0 done | 132 left)
30-Apr-2017 01:22:34 mask_areas                            finished   (0 run | 0 fail | 1 done | 132 left)
30-Apr-2017 01:22:34 tseries_subject1_session1_motor       submitted  (1 run | 0 fail | 1 done | 131 left)
30-Apr-2017 01:22:34 tseries_subject2_session1_motor       submitted  (2 run | 0 fail | 1 done | 130 left)
30-Apr-2017 01:22:34 neighbourhood_areas                   submitted  (3 run | 0 fail | 1 done | 129 left)
30-Apr-2017 01:22:36 tseries_subject1_session1_motor       finished   (2 run | 0 fail | 2 done | 129 left)
30-Apr-2017 01:22:37 tseries_subject2_session1_motor       finished   (1 run | 0 fail | 3 done | 129 left)
30-Apr-2017 01:22:37 neighbourhood_areas                   finished   (0 run | 0 fail | 4 done | 129 left)
30-Apr-2017 01:22:37 region_growing_area_2001              submitted  (1 run | 0 fail | 4 done | 128 left)
30-Apr-2017 01:22:37 region_growing_area_2002              submitted  (2 run | 0 fail | 4 done | 127 left)
30-Apr-2017 01:22:37 region_growing_area_2101              submitted  (3 run | 0 fail | 4 done | 126 left)
30-Apr-2017 01:22:37 region_growing_area_2102              submitted  (4 run | 0 fail | 4 done | 125 left)
30-Apr-2017 01:22:37 region_growing_area_2111              submitted  (5 run | 0 fail | 4 done | 124 left)
30-Apr-2017 01:22:37 region_growing_area_2112              submitted  (6 run | 0 fail | 4 done | 123 left)
30-Apr-2017 01:22:37 region_growing_area_2201              submitted  (7 run | 0 fail | 4 done | 122 left)
30-Apr-2017 01:22:37 region_growing_area_2202              submitted  (8 run | 0 fail | 4 done | 121 left)
30-Apr-2017 01:22:37 region_growing_area_2001              finished   (7 run | 0 fail | 5 done | 121 left)
30-Apr-2017 01:22:37 region_growing_area_2002              finished   (6 run | 0 fail | 6 done | 121 left)
30-Apr-2017 01:22:37 region_growing_area_2101              finished   (5 run | 0 fail | 7 done | 121 left)
30-Apr-2017 01:22:37 region_growing_area_2211              submitted  (6 run | 0 fail | 7 done | 120 left)
30-Apr-2017 01:22:37 region_growing_area_2212              submitted  (7 run | 0 fail | 7 done | 119 left)
30-Apr-2017 01:22:37 region_growing_area_2301              submitted  (8 run | 0 fail | 7 done | 118 left)
30-Apr-2017 01:22:38 region_growing_area_2102              finished   (7 run | 0 fail | 8 done | 118 left)
30-Apr-2017 01:22:38 region_growing_area_2111              finished   (6 run | 0 fail | 9 done | 118 left)
30-Apr-2017 01:22:38 region_growing_area_2201              finished   (5 run | 0 fail | 10 done | 118 left)
30-Apr-2017 01:22:38 region_growing_area_2301              finished   (4 run | 0 fail | 11 done | 118 left)
30-Apr-2017 01:22:38 region_growing_area_2302              submitted  (5 run | 0 fail | 11 done | 117 left)
30-Apr-2017 01:22:38 region_growing_area_2311              submitted  (6 run | 0 fail | 11 done | 116 left)
30-Apr-2017 01:22:38 region_growing_area_2312              submitted  (7 run | 0 fail | 11 done | 115 left)
30-Apr-2017 01:22:38 region_growing_area_2321              submitted  (8 run | 0 fail | 11 done | 114 left)
30-Apr-2017 01:22:38 region_growing_area_2202              finished   (7 run | 0 fail | 12 done | 114 left)
30-Apr-2017 01:22:38 region_growing_area_2211              finished   (6 run | 0 fail | 13 done | 114 left)
30-Apr-2017 01:22:38 region_growing_area_2112              finished   (5 run | 0 fail | 14 done | 114 left)
30-Apr-2017 01:22:38 region_growing_area_2212              finished   (4 run | 0 fail | 15 done | 114 left)
30-Apr-2017 01:22:38 region_growing_area_2302              finished   (3 run | 0 fail | 16 done | 114 left)
30-Apr-2017 01:22:38 region_growing_area_2311              finished   (2 run | 0 fail | 17 done | 114 left)
30-Apr-2017 01:22:38 region_growing_area_2312              finished   (1 run | 0 fail | 18 done | 114 left)
30-Apr-2017 01:22:38 region_growing_area_2321              finished   (0 run | 0 fail | 19 done | 114 left)
30-Apr-2017 01:22:38 region_growing_area_2322              submitted  (1 run | 0 fail | 19 done | 113 left)
30-Apr-2017 01:22:38 region_growing_area_2331              submitted  (2 run | 0 fail | 19 done | 112 left)
30-Apr-2017 01:22:38 region_growing_area_2332              submitted  (3 run | 0 fail | 19 done | 111 left)
30-Apr-2017 01:22:38 region_growing_area_2401              submitted  (4 run | 0 fail | 19 done | 110 left)
30-Apr-2017 01:22:38 region_growing_area_2402              submitted  (5 run | 0 fail | 19 done | 109 left)
30-Apr-2017 01:22:38 region_growing_area_2501              submitted  (6 run | 0 fail | 19 done | 108 left)
30-Apr-2017 01:22:38 region_growing_area_2502              submitted  (7 run | 0 fail | 19 done | 107 left)
30-Apr-2017 01:22:38 region_growing_area_2601              submitted  (8 run | 0 fail | 19 done | 106 left)
30-Apr-2017 01:22:39 region_growing_area_2322              finished   (7 run | 0 fail | 20 done | 106 left)
30-Apr-2017 01:22:39 region_growing_area_2332              finished   (6 run | 0 fail | 21 done | 106 left)
30-Apr-2017 01:22:39 region_growing_area_2401              finished   (5 run | 0 fail | 22 done | 106 left)
30-Apr-2017 01:22:39 region_growing_area_2602              submitted  (6 run | 0 fail | 22 done | 105 left)
30-Apr-2017 01:22:39 region_growing_area_2611              submitted  (7 run | 0 fail | 22 done | 104 left)
30-Apr-2017 01:22:39 region_growing_area_2612              submitted  (8 run | 0 fail | 22 done | 103 left)
30-Apr-2017 01:22:39 region_growing_area_2331              finished   (7 run | 0 fail | 23 done | 103 left)
30-Apr-2017 01:22:39 region_growing_area_2402              finished   (6 run | 0 fail | 24 done | 103 left)
30-Apr-2017 01:22:39 region_growing_area_2601              finished   (5 run | 0 fail | 25 done | 103 left)
30-Apr-2017 01:22:39 region_growing_area_2701              submitted  (6 run | 0 fail | 25 done | 102 left)
30-Apr-2017 01:22:39 region_growing_area_2702              submitted  (7 run | 0 fail | 25 done | 101 left)
30-Apr-2017 01:22:39 region_growing_area_2502              finished   (6 run | 0 fail | 26 done | 101 left)
30-Apr-2017 01:22:39 region_growing_area_3001              submitted  (7 run | 0 fail | 26 done | 100 left)
30-Apr-2017 01:22:40 region_growing_area_2501              finished   (6 run | 0 fail | 27 done | 100 left)
30-Apr-2017 01:22:40 region_growing_area_2602              finished   (5 run | 0 fail | 28 done | 100 left)
30-Apr-2017 01:22:40 region_growing_area_2611              finished   (4 run | 0 fail | 29 done | 100 left)
30-Apr-2017 01:22:40 region_growing_area_2612              finished   (3 run | 0 fail | 30 done | 100 left)
30-Apr-2017 01:22:40 region_growing_area_3002              submitted  (4 run | 0 fail | 30 done | 99 left)
30-Apr-2017 01:22:40 region_growing_area_4001              submitted  (5 run | 0 fail | 30 done | 98 left)
30-Apr-2017 01:22:40 region_growing_area_4002              submitted  (6 run | 0 fail | 30 done | 97 left)
30-Apr-2017 01:22:40 region_growing_area_4011              submitted  (7 run | 0 fail | 30 done | 96 left)
30-Apr-2017 01:22:40 region_growing_area_4012              submitted  (8 run | 0 fail | 30 done | 95 left)
30-Apr-2017 01:22:40 region_growing_area_2701              finished   (7 run | 0 fail | 31 done | 95 left)
30-Apr-2017 01:22:40 region_growing_area_2702              finished   (6 run | 0 fail | 32 done | 95 left)
30-Apr-2017 01:22:40 region_growing_area_3001              finished   (5 run | 0 fail | 33 done | 95 left)
30-Apr-2017 01:22:40 region_growing_area_3002              finished   (4 run | 0 fail | 34 done | 95 left)
30-Apr-2017 01:22:40 region_growing_area_4001              finished   (3 run | 0 fail | 35 done | 95 left)
30-Apr-2017 01:22:40 region_growing_area_4011              finished   (2 run | 0 fail | 36 done | 95 left)
30-Apr-2017 01:22:40 region_growing_area_4012              finished   (1 run | 0 fail | 37 done | 95 left)
30-Apr-2017 01:22:40 region_growing_area_4021              submitted  (2 run | 0 fail | 37 done | 94 left)
30-Apr-2017 01:22:40 region_growing_area_4022              submitted  (3 run | 0 fail | 37 done | 93 left)
30-Apr-2017 01:22:40 region_growing_area_4101              submitted  (4 run | 0 fail | 37 done | 92 left)
30-Apr-2017 01:22:40 region_growing_area_4102              submitted  (5 run | 0 fail | 37 done | 91 left)
30-Apr-2017 01:22:40 region_growing_area_4111              submitted  (6 run | 0 fail | 37 done | 90 left)
30-Apr-2017 01:22:40 region_growing_area_4112              submitted  (7 run | 0 fail | 37 done | 89 left)
30-Apr-2017 01:22:40 region_growing_area_4201              submitted  (8 run | 0 fail | 37 done | 88 left)
30-Apr-2017 01:22:41 region_growing_area_4002              finished   (7 run | 0 fail | 38 done | 88 left)
30-Apr-2017 01:22:41 region_growing_area_4022              finished   (6 run | 0 fail | 39 done | 88 left)
30-Apr-2017 01:22:41 region_growing_area_4101              finished   (5 run | 0 fail | 40 done | 88 left)
30-Apr-2017 01:22:41 region_growing_area_4111              finished   (4 run | 0 fail | 41 done | 88 left)
30-Apr-2017 01:22:41 region_growing_area_4202              submitted  (5 run | 0 fail | 41 done | 87 left)
30-Apr-2017 01:22:41 region_growing_area_5001              submitted  (6 run | 0 fail | 41 done | 86 left)
30-Apr-2017 01:22:41 region_growing_area_5002              submitted  (7 run | 0 fail | 41 done | 85 left)
30-Apr-2017 01:22:41 region_growing_area_5011              submitted  (8 run | 0 fail | 41 done | 84 left)
30-Apr-2017 01:22:42 region_growing_area_4021              finished   (7 run | 0 fail | 42 done | 84 left)
30-Apr-2017 01:22:42 region_growing_area_4112              finished   (6 run | 0 fail | 43 done | 84 left)
30-Apr-2017 01:22:42 region_growing_area_4201              finished   (5 run | 0 fail | 44 done | 84 left)
30-Apr-2017 01:22:42 region_growing_area_4202              finished   (4 run | 0 fail | 45 done | 84 left)
30-Apr-2017 01:22:42 region_growing_area_5012              submitted  (5 run | 0 fail | 45 done | 83 left)
30-Apr-2017 01:22:42 region_growing_area_5021              submitted  (6 run | 0 fail | 45 done | 82 left)
30-Apr-2017 01:22:42 region_growing_area_5022              submitted  (7 run | 0 fail | 45 done | 81 left)
30-Apr-2017 01:22:42 region_growing_area_5101              submitted  (8 run | 0 fail | 45 done | 80 left)
30-Apr-2017 01:22:42 region_growing_area_4102              finished   (7 run | 0 fail | 46 done | 80 left)
30-Apr-2017 01:22:42 region_growing_area_5001              finished   (6 run | 0 fail | 47 done | 80 left)
30-Apr-2017 01:22:42 region_growing_area_5011              finished   (5 run | 0 fail | 48 done | 80 left)
30-Apr-2017 01:22:42 region_growing_area_5002              finished   (4 run | 0 fail | 49 done | 80 left)
30-Apr-2017 01:22:42 region_growing_area_5012              finished   (3 run | 0 fail | 50 done | 80 left)
30-Apr-2017 01:22:42 region_growing_area_5021              finished   (2 run | 0 fail | 51 done | 80 left)
30-Apr-2017 01:22:42 region_growing_area_5022              finished   (1 run | 0 fail | 52 done | 80 left)
30-Apr-2017 01:22:42 region_growing_area_5101              finished   (0 run | 0 fail | 53 done | 80 left)
30-Apr-2017 01:22:42 region_growing_area_5102              submitted  (1 run | 0 fail | 53 done | 79 left)
30-Apr-2017 01:22:42 region_growing_area_5201              submitted  (2 run | 0 fail | 53 done | 78 left)
30-Apr-2017 01:22:42 region_growing_area_5202              submitted  (3 run | 0 fail | 53 done | 77 left)
30-Apr-2017 01:22:42 region_growing_area_5301              submitted  (4 run | 0 fail | 53 done | 76 left)
30-Apr-2017 01:22:42 region_growing_area_5302              submitted  (5 run | 0 fail | 53 done | 75 left)
30-Apr-2017 01:22:42 region_growing_area_5401              submitted  (6 run | 0 fail | 53 done | 74 left)
30-Apr-2017 01:22:42 region_growing_area_5402              submitted  (7 run | 0 fail | 53 done | 73 left)
30-Apr-2017 01:22:42 region_growing_area_6001              submitted  (8 run | 0 fail | 53 done | 72 left)
30-Apr-2017 01:22:43 region_growing_area_5102              finished   (7 run | 0 fail | 54 done | 72 left)
30-Apr-2017 01:22:43 region_growing_area_5201              finished   (6 run | 0 fail | 55 done | 72 left)
30-Apr-2017 01:22:43 region_growing_area_5202              finished   (5 run | 0 fail | 56 done | 72 left)
30-Apr-2017 01:22:43 region_growing_area_5301              finished   (4 run | 0 fail | 57 done | 72 left)
30-Apr-2017 01:22:43 region_growing_area_5402              finished   (3 run | 0 fail | 58 done | 72 left)
30-Apr-2017 01:22:43 region_growing_area_6002              submitted  (4 run | 0 fail | 58 done | 71 left)
30-Apr-2017 01:22:43 region_growing_area_6101              submitted  (5 run | 0 fail | 58 done | 70 left)
30-Apr-2017 01:22:43 region_growing_area_6102              submitted  (6 run | 0 fail | 58 done | 69 left)
30-Apr-2017 01:22:43 region_growing_area_6201              submitted  (7 run | 0 fail | 58 done | 68 left)
30-Apr-2017 01:22:43 region_growing_area_6202              submitted  (8 run | 0 fail | 58 done | 67 left)
30-Apr-2017 01:22:44 region_growing_area_5302              finished   (7 run | 0 fail | 59 done | 67 left)
30-Apr-2017 01:22:44 region_growing_area_5401              finished   (6 run | 0 fail | 60 done | 67 left)
30-Apr-2017 01:22:44 region_growing_area_6001              finished   (5 run | 0 fail | 61 done | 67 left)
30-Apr-2017 01:22:44 region_growing_area_6002              finished   (4 run | 0 fail | 62 done | 67 left)
30-Apr-2017 01:22:44 region_growing_area_6101              finished   (3 run | 0 fail | 63 done | 67 left)
30-Apr-2017 01:22:44 region_growing_area_6202              finished   (2 run | 0 fail | 64 done | 67 left)
30-Apr-2017 01:22:44 region_growing_area_6211              submitted  (3 run | 0 fail | 64 done | 66 left)
30-Apr-2017 01:22:44 region_growing_area_6212              submitted  (4 run | 0 fail | 64 done | 65 left)
30-Apr-2017 01:22:44 region_growing_area_6221              submitted  (5 run | 0 fail | 64 done | 64 left)
30-Apr-2017 01:22:44 region_growing_area_6222              submitted  (6 run | 0 fail | 64 done | 63 left)
30-Apr-2017 01:22:44 region_growing_area_6301              submitted  (7 run | 0 fail | 64 done | 62 left)
30-Apr-2017 01:22:44 region_growing_area_6302              submitted  (8 run | 0 fail | 64 done | 61 left)
30-Apr-2017 01:22:44 region_growing_area_6201              finished   (7 run | 0 fail | 65 done | 61 left)
30-Apr-2017 01:22:44 region_growing_area_6102              finished   (6 run | 0 fail | 66 done | 61 left)
30-Apr-2017 01:22:44 region_growing_area_6211              finished   (5 run | 0 fail | 67 done | 61 left)
30-Apr-2017 01:22:44 region_growing_area_6212              finished   (4 run | 0 fail | 68 done | 61 left)
30-Apr-2017 01:22:44 region_growing_area_6222              finished   (3 run | 0 fail | 69 done | 61 left)
30-Apr-2017 01:22:44 region_growing_area_6301              finished   (2 run | 0 fail | 70 done | 61 left)
30-Apr-2017 01:22:44 region_growing_area_6401              submitted  (3 run | 0 fail | 70 done | 60 left)
30-Apr-2017 01:22:44 region_growing_area_6402              submitted  (4 run | 0 fail | 70 done | 59 left)
30-Apr-2017 01:22:44 region_growing_area_7001              submitted  (5 run | 0 fail | 70 done | 58 left)
30-Apr-2017 01:22:44 region_growing_area_7002              submitted  (6 run | 0 fail | 70 done | 57 left)
30-Apr-2017 01:22:44 region_growing_area_7011              submitted  (7 run | 0 fail | 70 done | 56 left)
30-Apr-2017 01:22:44 region_growing_area_7012              submitted  (8 run | 0 fail | 70 done | 55 left)
30-Apr-2017 01:22:44 region_growing_area_6221              finished   (7 run | 0 fail | 71 done | 55 left)
30-Apr-2017 01:22:45 region_growing_area_6302              finished   (6 run | 0 fail | 72 done | 55 left)
30-Apr-2017 01:22:45 region_growing_area_6401              finished   (5 run | 0 fail | 73 done | 55 left)
30-Apr-2017 01:22:45 region_growing_area_6402              finished   (4 run | 0 fail | 74 done | 55 left)
30-Apr-2017 01:22:45 region_growing_area_7001              finished   (3 run | 0 fail | 75 done | 55 left)
30-Apr-2017 01:22:45 region_growing_area_7011              finished   (2 run | 0 fail | 76 done | 55 left)
30-Apr-2017 01:22:45 region_growing_area_7012              finished   (1 run | 0 fail | 77 done | 55 left)
30-Apr-2017 01:22:45 region_growing_area_7021              submitted  (2 run | 0 fail | 77 done | 54 left)
30-Apr-2017 01:22:45 region_growing_area_7022              submitted  (3 run | 0 fail | 77 done | 53 left)
30-Apr-2017 01:22:45 region_growing_area_7101              submitted  (4 run | 0 fail | 77 done | 52 left)
30-Apr-2017 01:22:45 region_growing_area_7102              submitted  (5 run | 0 fail | 77 done | 51 left)
30-Apr-2017 01:22:45 region_growing_area_8101              submitted  (6 run | 0 fail | 77 done | 50 left)
30-Apr-2017 01:22:45 region_growing_area_8102              submitted  (7 run | 0 fail | 77 done | 49 left)
30-Apr-2017 01:22:45 region_growing_area_8111              submitted  (8 run | 0 fail | 77 done | 48 left)
30-Apr-2017 01:22:45 region_growing_area_7002              finished   (7 run | 0 fail | 78 done | 48 left)
30-Apr-2017 01:22:45 region_growing_area_7022              finished   (6 run | 0 fail | 79 done | 48 left)
30-Apr-2017 01:22:45 region_growing_area_7101              finished   (5 run | 0 fail | 80 done | 48 left)
30-Apr-2017 01:22:45 region_growing_area_8112              submitted  (6 run | 0 fail | 80 done | 47 left)
30-Apr-2017 01:22:45 region_growing_area_8121              submitted  (7 run | 0 fail | 80 done | 46 left)
30-Apr-2017 01:22:45 region_growing_area_8122              submitted  (8 run | 0 fail | 80 done | 45 left)
30-Apr-2017 01:22:45 region_growing_area_8111              finished   (7 run | 0 fail | 81 done | 45 left)
30-Apr-2017 01:22:46 region_growing_area_7021              finished   (6 run | 0 fail | 82 done | 45 left)
30-Apr-2017 01:22:46 region_growing_area_7102              finished   (5 run | 0 fail | 83 done | 45 left)
30-Apr-2017 01:22:46 region_growing_area_8101              finished   (4 run | 0 fail | 84 done | 45 left)
30-Apr-2017 01:22:46 region_growing_area_8102              finished   (3 run | 0 fail | 85 done | 45 left)
30-Apr-2017 01:22:46 region_growing_area_8112              finished   (2 run | 0 fail | 86 done | 45 left)
30-Apr-2017 01:22:46 region_growing_area_8122              finished   (1 run | 0 fail | 87 done | 45 left)
30-Apr-2017 01:22:46 region_growing_area_8201              submitted  (2 run | 0 fail | 87 done | 44 left)
30-Apr-2017 01:22:46 region_growing_area_8202              submitted  (3 run | 0 fail | 87 done | 43 left)
30-Apr-2017 01:22:46 region_growing_area_8211              submitted  (4 run | 0 fail | 87 done | 42 left)
30-Apr-2017 01:22:46 region_growing_area_8212              submitted  (5 run | 0 fail | 87 done | 41 left)
30-Apr-2017 01:22:46 region_growing_area_8301              submitted  (6 run | 0 fail | 87 done | 40 left)
30-Apr-2017 01:22:46 region_growing_area_8302              submitted  (7 run | 0 fail | 87 done | 39 left)
30-Apr-2017 01:22:46 region_growing_area_9001              submitted  (8 run | 0 fail | 87 done | 38 left)
30-Apr-2017 01:22:46 region_growing_area_8121              finished   (7 run | 0 fail | 88 done | 38 left)
30-Apr-2017 01:22:47 region_growing_area_8201              finished   (6 run | 0 fail | 89 done | 38 left)
30-Apr-2017 01:22:47 region_growing_area_8202              finished   (5 run | 0 fail | 90 done | 38 left)
30-Apr-2017 01:22:47 region_growing_area_8211              finished   (4 run | 0 fail | 91 done | 38 left)
30-Apr-2017 01:22:47 region_growing_area_9002              submitted  (5 run | 0 fail | 91 done | 37 left)
30-Apr-2017 01:22:47 region_growing_area_9011              submitted  (6 run | 0 fail | 91 done | 36 left)
30-Apr-2017 01:22:47 region_growing_area_9012              submitted  (7 run | 0 fail | 91 done | 35 left)
30-Apr-2017 01:22:47 region_growing_area_9021              submitted  (8 run | 0 fail | 91 done | 34 left)
30-Apr-2017 01:22:47 region_growing_area_8302              finished   (7 run | 0 fail | 92 done | 34 left)
30-Apr-2017 01:22:47 region_growing_area_9001              finished   (6 run | 0 fail | 93 done | 34 left)
30-Apr-2017 01:22:47 region_growing_area_8301              finished   (5 run | 0 fail | 94 done | 34 left)
30-Apr-2017 01:22:47 region_growing_area_8212              finished   (4 run | 0 fail | 95 done | 34 left)
30-Apr-2017 01:22:47 region_growing_area_9002              finished   (3 run | 0 fail | 96 done | 34 left)
30-Apr-2017 01:22:47 region_growing_area_9011              finished   (2 run | 0 fail | 97 done | 34 left)
30-Apr-2017 01:22:47 region_growing_area_9012              finished   (1 run | 0 fail | 98 done | 34 left)
30-Apr-2017 01:22:47 region_growing_area_9022              submitted  (2 run | 0 fail | 98 done | 33 left)
30-Apr-2017 01:22:47 region_growing_area_9031              submitted  (3 run | 0 fail | 98 done | 32 left)
30-Apr-2017 01:22:47 region_growing_area_9032              submitted  (4 run | 0 fail | 98 done | 31 left)
30-Apr-2017 01:22:47 region_growing_area_9041              submitted  (5 run | 0 fail | 98 done | 30 left)
30-Apr-2017 01:22:47 region_growing_area_9042              submitted  (6 run | 0 fail | 98 done | 29 left)
30-Apr-2017 01:22:47 region_growing_area_9051              submitted  (7 run | 0 fail | 98 done | 28 left)
30-Apr-2017 01:22:47 region_growing_area_9052              submitted  (8 run | 0 fail | 98 done | 27 left)
30-Apr-2017 01:22:47 region_growing_area_9021              finished   (7 run | 0 fail | 99 done | 27 left)
30-Apr-2017 01:22:48 region_growing_area_9022              finished   (6 run | 0 fail | 100 done | 27 left)
30-Apr-2017 01:22:48 region_growing_area_9031              finished   (5 run | 0 fail | 101 done | 27 left)
30-Apr-2017 01:22:48 region_growing_area_9032              finished   (4 run | 0 fail | 102 done | 27 left)
30-Apr-2017 01:22:48 region_growing_area_9042              finished   (3 run | 0 fail | 103 done | 27 left)
30-Apr-2017 01:22:48 region_growing_area_9061              submitted  (4 run | 0 fail | 103 done | 26 left)
30-Apr-2017 01:22:48 region_growing_area_9062              submitted  (5 run | 0 fail | 103 done | 25 left)
30-Apr-2017 01:22:48 region_growing_area_9071              submitted  (6 run | 0 fail | 103 done | 24 left)
30-Apr-2017 01:22:48 region_growing_area_9072              submitted  (7 run | 0 fail | 103 done | 23 left)
30-Apr-2017 01:22:48 region_growing_area_9081              submitted  (8 run | 0 fail | 103 done | 22 left)
30-Apr-2017 01:22:48 region_growing_area_9041              finished   (7 run | 0 fail | 104 done | 22 left)
30-Apr-2017 01:22:48 region_growing_area_9082              submitted  (8 run | 0 fail | 104 done | 21 left)
30-Apr-2017 01:22:49 region_growing_area_9051              finished   (7 run | 0 fail | 105 done | 21 left)
30-Apr-2017 01:22:49 region_growing_area_9052              finished   (6 run | 0 fail | 106 done | 21 left)
30-Apr-2017 01:22:49 region_growing_area_9061              finished   (5 run | 0 fail | 107 done | 21 left)
30-Apr-2017 01:22:49 region_growing_area_9062              finished   (4 run | 0 fail | 108 done | 21 left)
30-Apr-2017 01:22:49 region_growing_area_9071              finished   (3 run | 0 fail | 109 done | 21 left)
30-Apr-2017 01:22:49 region_growing_area_9072              finished   (2 run | 0 fail | 110 done | 21 left)
30-Apr-2017 01:22:49 region_growing_area_9081              finished   (1 run | 0 fail | 111 done | 21 left)
30-Apr-2017 01:22:49 region_growing_area_9100              submitted  (2 run | 0 fail | 111 done | 20 left)
30-Apr-2017 01:22:49 region_growing_area_9110              submitted  (3 run | 0 fail | 111 done | 19 left)
30-Apr-2017 01:22:49 region_growing_area_9120              submitted  (4 run | 0 fail | 111 done | 18 left)
30-Apr-2017 01:22:49 region_growing_area_9130              submitted  (5 run | 0 fail | 111 done | 17 left)
30-Apr-2017 01:22:49 region_growing_area_9140              submitted  (6 run | 0 fail | 111 done | 16 left)
30-Apr-2017 01:22:49 region_growing_area_9150              submitted  (7 run | 0 fail | 111 done | 15 left)
30-Apr-2017 01:22:49 region_growing_area_9160              submitted  (8 run | 0 fail | 111 done | 14 left)
30-Apr-2017 01:22:49 region_growing_area_9082              finished   (7 run | 0 fail | 112 done | 14 left)
30-Apr-2017 01:22:49 region_growing_area_9100              finished   (6 run | 0 fail | 113 done | 14 left)
30-Apr-2017 01:22:49 region_growing_area_9120              finished   (5 run | 0 fail | 114 done | 14 left)
30-Apr-2017 01:22:49 region_growing_area_9170              submitted  (6 run | 0 fail | 114 done | 13 left)
30-Apr-2017 01:22:49 region_growing_area_9110              finished   (5 run | 0 fail | 115 done | 13 left)
30-Apr-2017 01:22:49 region_growing_area_9140              finished   (4 run | 0 fail | 116 done | 13 left)
30-Apr-2017 01:22:50 region_growing_area_9130              finished   (3 run | 0 fail | 117 done | 13 left)
30-Apr-2017 01:22:50 region_growing_area_9150              finished   (2 run | 0 fail | 118 done | 13 left)
30-Apr-2017 01:22:50 region_growing_area_9160              finished   (1 run | 0 fail | 119 done | 13 left)
30-Apr-2017 01:22:50 region_growing_area_9170              finished   (0 run | 0 fail | 120 done | 13 left)
30-Apr-2017 01:22:50 merge_part                            submitted  (1 run | 0 fail | 120 done | 12 left)
30-Apr-2017 01:22:51 merge_part                            finished   (0 run | 0 fail | 121 done | 12 left)
30-Apr-2017 01:22:51 tseries_atoms_subject1_session1_motor submitted  (1 run | 0 fail | 121 done | 11 left)
30-Apr-2017 01:22:51 tseries_atoms_subject2_session1_motor submitted  (2 run | 0 fail | 121 done | 10 left)
30-Apr-2017 01:22:52 tseries_atoms_subject1_session1_motor finished   (1 run | 0 fail | 122 done | 10 left)
30-Apr-2017 01:22:52 tseries_atoms_subject2_session1_motor finished   (0 run | 0 fail | 123 done | 10 left)
30-Apr-2017 01:22:52 stability_ind_subject1                submitted  (1 run | 0 fail | 123 done | 9 left)
30-Apr-2017 01:22:52 stability_ind_subject2                submitted  (2 run | 0 fail | 123 done | 8 left)
.
30-Apr-2017 01:23:32 stability_ind_subject1                finished   (1 run | 0 fail | 124 done | 8 left)
30-Apr-2017 01:23:32 msteps_ind_subject1                   submitted  (2 run | 0 fail | 124 done | 7 left)
30-Apr-2017 01:23:33 msteps_ind_subject1                   finished   (1 run | 0 fail | 125 done | 7 left)
.
30-Apr-2017 01:23:40 stability_ind_subject2                finished   (0 run | 0 fail | 126 done | 7 left)
30-Apr-2017 01:23:40 stability_group_sci10                 submitted  (1 run | 0 fail | 126 done | 6 left)
30-Apr-2017 01:23:40 stability_group_sci20                 submitted  (2 run | 0 fail | 126 done | 5 left)
30-Apr-2017 01:23:40 stability_group_sci30                 submitted  (3 run | 0 fail | 126 done | 4 left)
30-Apr-2017 01:23:40 summary_stability_avg_ind             submitted  (4 run | 0 fail | 126 done | 3 left)
30-Apr-2017 01:23:40 msteps_ind_subject2                   submitted  (5 run | 0 fail | 126 done | 2 left)
30-Apr-2017 01:23:42 msteps_ind_subject2                   finished   (4 run | 0 fail | 127 done | 2 left)
30-Apr-2017 01:23:45 summary_stability_avg_ind             finished   (3 run | 0 fail | 128 done | 2 left)
30-Apr-2017 01:23:45 Stopping idle worker 4 (not enough jobs left to do).
.
30-Apr-2017 01:24:33 stability_group_sci20                 finished   (2 run | 0 fail | 129 done | 2 left)
30-Apr-2017 01:24:33 Stopping idle worker 2 (not enough jobs left to do).
30-Apr-2017 01:24:36 stability_group_sci10                 finished   (1 run | 0 fail | 130 done | 2 left)
30-Apr-2017 01:24:36 stability_group_sci30                 finished   (0 run | 0 fail | 131 done | 2 left)
30-Apr-2017 01:24:36 summary_stability_group               submitted  (1 run | 0 fail | 131 done | 1 left)
30-Apr-2017 01:24:36 msteps_group                          submitted  (2 run | 0 fail | 131 done | 0 left)
.
30-Apr-2017 01:24:40 summary_stability_group               finished   (1 run | 0 fail | 132 done | 0 left)
30-Apr-2017 01:24:40 Stopping idle worker 1 (not enough jobs left to do).
Deamon terminated on 30-Apr-2017 01:24:51

30-Apr-2017 01:24:51 msteps_group                          finished   (0 run | 0 fail | 133 done | 0 left)
30-Apr-2017 01:24:51 Stopping idle worker 3 (not enough jobs left to do).

*******************************************
Pipeline terminated on 30-Apr-2017 01:24:51
All jobs have been successfully completed.

Now that the pipeline has successfully completed, we can check a .csv file generated as part of the outpus. This .csv file list the number of clusters identified as "representative" of all stable solutions across the grid of clustering parameters we selected, using a method called MSTEPS.

In [10]:
file_msteps = [opt.folder_out filesep 'stability_group' filesep 'msteps_group_table.csv'];
[tab,lx,ly] = niak_read_csv(file_msteps);
tab
tab =

   10    7    8
   10   12   11
   20   16   22
   20   24   27
   30   30   40

So each line is a number of clusters at the individual/group/consensus levels. Here MSTEPS has selected three scales as appropriate to summarize all tested numbers of clusters. Note that, even though we selected only 10, 20 and 30 in our grid, that grid is strictly used only at the individual level. At the group and consensus levels, other parameters are explored, in the neighbourhood of the number of individual clusters. We can now set these parameters in the pipeline to generate stability maps and parcellations:

In [11]:
opt.scales_maps = tab;

We can finally restart the pipeline. It will pick up where it left, and only deal with the generation of the maps.

In [12]:
niak_pipeline_stability_rest(files_in,opt);
Logs will be stored in /sandbox/home/git/niak_tutorials/basc_test_niak/logs/
Generating dependencies ...
   Percentage completed :  0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100- 0.35 sec
Setting up the to-do list ...
   I found 10 job(s) to do, and 133 job(s) already completed.

Deamon started on 30-Apr-2017 01:25:00
30-Apr-2017 01:25:00 Starting the pipeline manager...
30-Apr-2017 01:25:00 Starting the garbage collector...
30-Apr-2017 01:25:00 Starting worker number 1...
30-Apr-2017 01:25:00 Starting worker number 2...
30-Apr-2017 01:25:01 Starting worker number 3...
30-Apr-2017 01:25:01 Starting worker number 4...

Pipeline started on 30-Apr-2017 01:25:02
user: , host: 3f86dc5d68dc, system: unix
****************************************
30-Apr-2017 01:25:05 stability_maps_group_sci10_scg7_scf8     submitted  (1 run | 0 fail | 133 done | 9 left)
30-Apr-2017 01:25:05 figure_stability_group_sci10_scg7_scf8   submitted  (2 run | 0 fail | 133 done | 8 left)
30-Apr-2017 01:25:05 stability_maps_group_sci10_scg12_scf11   submitted  (3 run | 0 fail | 133 done | 7 left)
30-Apr-2017 01:25:05 figure_stability_group_sci10_scg12_scf11 submitted  (4 run | 0 fail | 133 done | 6 left)
30-Apr-2017 01:25:05 stability_maps_group_sci20_scg16_scf22   submitted  (5 run | 0 fail | 133 done | 5 left)
30-Apr-2017 01:25:06 figure_stability_group_sci20_scg16_scf22 submitted  (6 run | 0 fail | 133 done | 4 left)
30-Apr-2017 01:25:06 stability_maps_group_sci20_scg24_scf27   submitted  (7 run | 0 fail | 133 done | 3 left)
30-Apr-2017 01:25:06 figure_stability_group_sci20_scg24_scf27 submitted  (8 run | 0 fail | 133 done | 2 left)
30-Apr-2017 01:25:08 stability_maps_group_sci10_scg7_scf8     finished   (7 run | 0 fail | 134 done | 2 left)
30-Apr-2017 01:25:08 stability_maps_group_sci30_scg30_scf40   submitted  (8 run | 0 fail | 134 done | 1 left)
30-Apr-2017 01:25:08 stability_maps_group_sci10_scg12_scf11   finished   (7 run | 0 fail | 135 done | 1 left)
30-Apr-2017 01:25:08 figure_stability_group_sci30_scg30_scf40 submitted  (8 run | 0 fail | 135 done | 0 left)
30-Apr-2017 01:25:10 stability_maps_group_sci20_scg16_scf22   finished   (7 run | 0 fail | 136 done | 0 left)
30-Apr-2017 01:25:10 stability_maps_group_sci20_scg24_scf27   finished   (6 run | 0 fail | 137 done | 0 left)
30-Apr-2017 01:25:12 stability_maps_group_sci30_scg30_scf40   finished   (5 run | 0 fail | 138 done | 0 left)
30-Apr-2017 01:25:12 Stopping idle worker 1 (not enough jobs left to do).
30-Apr-2017 01:25:13 figure_stability_group_sci10_scg7_scf8   finished   (4 run | 0 fail | 139 done | 0 left)
30-Apr-2017 01:25:13 figure_stability_group_sci10_scg12_scf11 finished   (3 run | 0 fail | 140 done | 0 left)
30-Apr-2017 01:25:18 figure_stability_group_sci30_scg30_scf40 finished   (2 run | 0 fail | 141 done | 0 left)
30-Apr-2017 01:25:18 Stopping idle worker 3 (not enough jobs left to do).
30-Apr-2017 01:25:18 figure_stability_group_sci20_scg16_scf22 finished   (1 run | 0 fail | 142 done | 0 left)
30-Apr-2017 01:25:18 Stopping idle worker 2 (not enough jobs left to do).
Deamon terminated on 30-Apr-2017 01:25:19

30-Apr-2017 01:25:19 figure_stability_group_sci20_scg24_scf27 finished   (0 run | 0 fail | 143 done | 0 left)
30-Apr-2017 01:25:19 Stopping idle worker 4 (not enough jobs left to do).

*******************************************
Pipeline terminated on 30-Apr-2017 01:25:19
All jobs have been successfully completed.

The primary output of the BASC pipeline is a set of parcels for the selected pipeline. For example, the line [10 7 8] in opt.scales_maps means 10 individual clusters, 7 group clusters and 8 consensus (final) group clusters. Let's load and visualize that brain parcellation:

In [13]:
path_sc8 = [opt.folder_out filesep 'stability_group' filesep 'sci10_scg7_scf8' filesep];
parcel_sc8 =  [path_sc8 'brain_partition_consensus_group_sci10_scg7_scf8.nii.gz'];
[hdr,sc8] = niak_read_vol(parcel_sc8);
niak_montage(sc8)

On this small dataset with big voxels, the parcellation does not look good. But that is enough to illustrate how to run the pipeline... Other (longer) tutorials will show how to explore the results on a dataset with smaller voxels and more subjects.