The RMVA Inteface: TMVA and R

Required headers

In [1]:
#include "TRInterface.h"
#include "TMVA/MethodC50.h"
#include "TMVA/MethodRSNNS.h"
#include "TMVA/MethodRXGB.h"

Declare Factory

In [2]:
TMVA::Tools::Instance();

auto inputFile = TFile::Open("https://raw.githubusercontent.com/iml-wg/tmvatutorials/master/inputdata.root");
auto outputFile = TFile::Open("TMVAOutputCV.root", "RECREATE");

TMVA::Factory factory("TMVAClassification", outputFile,
                      "!V:ROC:!Correlations:!Silent:Color:!DrawProgressBar:AnalysisType=Classification" );
--- Factory                  : You are running ROOT Version: 6.07/07, Apr 1, 2016
--- Factory                  : 
--- Factory                  : _/_/_/_/_/ _|      _|  _|      _|    _|_|   
--- Factory                  :    _/      _|_|  _|_|  _|      _|  _|    _| 
--- Factory                  :   _/       _|  _|  _|  _|      _|  _|_|_|_| 
--- Factory                  :  _/        _|      _|    _|  _|    _|    _| 
--- Factory                  : _/         _|      _|      _|      _|    _| 
--- Factory                  : 
--- Factory                  : ___________TMVA Version 4.2.1, Feb 5, 2015
--- Factory                  : 

Declare DataLoader

In [3]:
TMVA::DataLoader loader("dataset");

//adding variables to dataset
loader.AddVariable("var1");
loader.AddVariable("var2");
loader.AddVariable("var3");
loader.AddVariable("var4");

Setting up Dataset

In [4]:
TTree *tsignal, *tbackground;
inputFile->GetObject("Sig", tsignal);
inputFile->GetObject("Bkg", tbackground);

TCut mycuts, mycutb;
   
loader.AddSignalTree     (tsignal, 1);      //signal weight = 1
loader.AddBackgroundTree (tbackground, 1);  //background weight = 1 

loader.PrepareTrainingAndTestTree(mycuts, mycutb,
"nTrain_Signal=1000:nTrain_Background=1000:SplitMode=Random:NormMode=NumEvents:!V");
--- DataSetInfo              : Dataset[dataset] : Added class "Signal"	 with internal class number 0
--- dataset                  : Add Tree Sig of type Signal with 6000 events
--- DataSetInfo              : Dataset[dataset] : Added class "Background"	 with internal class number 1
--- dataset                  : Add Tree Bkg of type Background with 6000 events
--- dataset                  : Preparing trees for training and testing...

Booking methods

The available Booking methods with options for RMVA are:

In [5]:
//C50 Boosted Decision Trees (BDTs)
factory.BookMethod(&loader, TMVA::Types::kC50, "C50",
                   "!H:NTrials=5:Rules=kTRUE:ControlSubSet=kFALSE:ControlBands=10:ControlWinnow=kFALSE:ControlNoGlobalPruning=kTRUE:ControlCF=0.25:ControlMinCases=2:ControlFuzzyThreshold=kTRUE:ControlSample=0:ControlEarlyStopping=kTRUE:!V" );
   
//Neural Networks using RSNNS package
factory.BookMethod(&loader, TMVA::Types::kRSNNS, "RMLP",
                   "!H:VarTransform=N:Size=c(5):Maxit=10:InitFunc=Randomize_Weights:LearnFunc=Std_Backpropagation:LearnFuncParams=c(0.2,0):!V" );

//eXtreme Gradient Boosted XGB Decision Trees
factory.BookMethod(&loader, TMVA::Types::kRXGB, "RXGB","!V:NRounds=20:MaxDepth=2:Eta=1" );

//TMVA BDTs
factory.BookMethod(&loader,TMVA::Types::kBDT, "BDT",
                   "!V:NTrees=50:MinNodeSize=2.5%:MaxDepth=2:BoostType=AdaBoost:AdaBoostBeta=0.5:UseBaggedBoost:BaggedSampleFraction=0.5:SeparationType=GiniIndex:nCuts=20" );
--- Factory                  : Booking method: C50 DataSet Name: dataset
--- DataSetFactory           : Dataset[dataset] : Splitmode is: "RANDOM" the mixmode is: "SAMEASSPLITMODE"
--- DataSetFactory           : Dataset[dataset] : Create training and testing trees -- looping over class "Signal" ...
--- DataSetFactory           : Dataset[dataset] : Weight expression for class 'Signal': ""
--- DataSetFactory           : Dataset[dataset] : Create training and testing trees -- looping over class "Background" ...
--- DataSetFactory           : Dataset[dataset] : Weight expression for class 'Background': ""
--- DataSetFactory           : Dataset[dataset] : Number of events in input trees (after possible flattening of arrays):
--- DataSetFactory           : Dataset[dataset] :     Signal          -- number of events       : 6000   / sum of weights: 6000 
--- DataSetFactory           : Dataset[dataset] :     Background      -- number of events       : 6000   / sum of weights: 6000 
--- DataSetFactory           : Dataset[dataset] :     Signal     tree -- total number of entries: 6000 
--- DataSetFactory           : Dataset[dataset] :     Background tree -- total number of entries: 6000 
--- DataSetFactory           : Dataset[dataset] : Preselection: (will NOT affect number of requested training and testing events)
--- DataSetFactory           : Dataset[dataset] :     No preselection cuts applied on event classes
--- DataSetFactory           : Dataset[dataset] : Weight renormalisation mode: "NumEvents": renormalises all event classes 
--- DataSetFactory           : Dataset[dataset] :  such that the effective (weighted) number of events in each class equals the respective 
--- DataSetFactory           : Dataset[dataset] :  number of events (entries) that you demanded in PrepareTrainingAndTestTree("","nTrain_Signal=.. )
--- DataSetFactory           : Dataset[dataset] :  ... i.e. such that Sum[i=1..N_j]{w_i} = N_j, j=0,1,2...
--- DataSetFactory           : Dataset[dataset] :  ... (note that N_j is the sum of TRAINING events (nTrain_j...with j=Signal,Background..
--- DataSetFactory           : Dataset[dataset] :  ..... Testing events are not renormalised nor included in the renormalisation factor! )
--- DataSetFactory           : Dataset[dataset] : --> Rescale Signal     event weights by factor: 1
--- DataSetFactory           : Dataset[dataset] : --> Rescale Background event weights by factor: 1
--- DataSetFactory           : Dataset[dataset] : Number of training and testing events after rescaling:
--- DataSetFactory           : Dataset[dataset] : ---------------------------------------------------------------------------
--- DataSetFactory           : Dataset[dataset] : Signal     -- training events            : 1000 (sum of weights: 1000) - requested were 1000 events
--- DataSetFactory           : Dataset[dataset] : Signal     -- testing events             : 5000 (sum of weights: 5000) - requested were 0 events
--- DataSetFactory           : Dataset[dataset] : Signal     -- training and testing events: 6000 (sum of weights: 6000)
--- DataSetFactory           : Dataset[dataset] : Background -- training events            : 1000 (sum of weights: 1000) - requested were 1000 events
--- DataSetFactory           : Dataset[dataset] : Background -- testing events             : 5000 (sum of weights: 5000) - requested were 0 events
--- DataSetFactory           : Dataset[dataset] : Background -- training and testing events: 6000 (sum of weights: 6000)
--- DataSetFactory           : Dataset[dataset] : Create internal training tree
--- DataSetFactory           : Dataset[dataset] : Create internal testing tree
--- DataSetInfo              : Dataset[dataset] : Correlation matrix (Signal):
--- DataSetInfo              : ----------------------------------------
--- DataSetInfo              :             var1    var2    var3    var4
--- DataSetInfo              :    var1:  +1.000  +0.386  +0.597  +0.808
--- DataSetInfo              :    var2:  +0.386  +1.000  +0.696  +0.743
--- DataSetInfo              :    var3:  +0.597  +0.696  +1.000  +0.860
--- DataSetInfo              :    var4:  +0.808  +0.743  +0.860  +1.000
--- DataSetInfo              : ----------------------------------------
--- DataSetInfo              : Dataset[dataset] : Correlation matrix (Background):
--- DataSetInfo              : ----------------------------------------
--- DataSetInfo              :             var1    var2    var3    var4
--- DataSetInfo              :    var1:  +1.000  +0.856  +0.914  +0.964
--- DataSetInfo              :    var2:  +0.856  +1.000  +0.927  +0.937
--- DataSetInfo              :    var3:  +0.914  +0.927  +1.000  +0.971
--- DataSetInfo              :    var4:  +0.964  +0.937  +0.971  +1.000
--- DataSetInfo              : ----------------------------------------
--- DataSetFactory           : Dataset[dataset] :  
--- Factory                  : Booking method: RMLP DataSet Name: dataset
--- RMLP                     : Dataset[dataset] : Create Transformation "N" with events from all classes.
--- Norm                     : Transformation, Variable selection : 
--- Norm                     : Input : variable 'var1' (index=0).   <---> Output : variable 'var1' (index=0).
--- Norm                     : Input : variable 'var2' (index=1).   <---> Output : variable 'var2' (index=1).
--- Norm                     : Input : variable 'var3' (index=2).   <---> Output : variable 'var3' (index=2).
--- Norm                     : Input : variable 'var4' (index=3).   <---> Output : variable 'var4' (index=3).
--- Factory                  : Booking method: RXGB DataSet Name: dataset
--- Factory                  : Booking method: BDT DataSet Name: dataset

Training the Methods

In [6]:
factory.TrainAllMethods();
--- Factory                  :  
--- Factory                  : Train all methods for Classification ...
--- Factory                  : 
--- Factory                  : current transformation string: 'I'
--- Factory                  : Dataset[dataset] : Create Transformation "I" with events from all classes.
--- Id                       : Transformation, Variable selection : 
--- Id                       : Input : variable 'var1' (index=0).   <---> Output : variable 'var1' (index=0).
--- Id                       : Input : variable 'var2' (index=1).   <---> Output : variable 'var2' (index=1).
--- Id                       : Input : variable 'var3' (index=2).   <---> Output : variable 'var3' (index=2).
--- Id                       : Input : variable 'var4' (index=3).   <---> Output : variable 'var4' (index=3).
--- Id                       : Preparing the Identity transformation...
--- TFHandler_Factory        : -----------------------------------------------------------
--- TFHandler_Factory        : Variable        Mean        RMS   [        Min        Max ]
--- TFHandler_Factory        : -----------------------------------------------------------
--- TFHandler_Factory        :     var1: 0.00077102     1.6695   [    -5.8991     4.7639 ]
--- TFHandler_Factory        :     var2: -0.0063164     1.5765   [    -5.2454     4.8300 ]
--- TFHandler_Factory        :     var3:  -0.010870     1.7365   [    -5.3563     4.6430 ]
--- TFHandler_Factory        :     var4:    0.14557     2.1608   [    -6.9675     5.0307 ]
--- TFHandler_Factory        : -----------------------------------------------------------
--- TFHandler_Factory        : Plot event variables for Id
--- TFHandler_Factory        : Create scatter and profile plots in target-file directory: 
--- TFHandler_Factory        : TMVAOutputCV.root:/dataset/InputVariables_Id/CorrelationPlots
--- TFHandler_Factory        :  
--- TFHandler_Factory        : Ranking input variables (method unspecific)...
--- IdTransformation         : Ranking result (top variable is best ranked)
--- IdTransformation         : -----------------------------
--- IdTransformation         : Rank : Variable  : Separation
--- IdTransformation         : -----------------------------
--- IdTransformation         :    1 : var4      : 3.458e-01
--- IdTransformation         :    2 : var3      : 2.817e-01
--- IdTransformation         :    3 : var1      : 2.640e-01
--- IdTransformation         :    4 : var2      : 2.173e-01
--- IdTransformation         : -----------------------------
--- Factory                  : Train method: C50 for Classification
--- C50                      : Dataset[dataset] : Begin training
--- C50                      : 
--- C50                      : --- Saving State File In:weights/C50Model.RData
--- C50                      : 
--- C50                      : Dataset[dataset] : End of training                                              
--- C50                      : Dataset[dataset] : Elapsed time for training with 2000 events: 0.873 sec         
--- C50                      : Dataset[dataset] : Create MVA output for Dataset[dataset] : classification on training sample
--- C50                      : Dataset[dataset] : Evaluation of C50 on training sample (2000 events)
--- C50                      : Dataset[dataset] : Elapsed time for evaluation of 2000 events: 0.0748 sec       
--- C50                      : Dataset[dataset] : Creating weight file in xml format: weights/TMVAClassification_C50.weights.xml
--- Factory                  : Training finished
--- Factory                  : Train method: RMLP for Classification
--- Norm                     : Preparing the transformation.
--- TFHandler_RMLP           : -----------------------------------------------------------
--- TFHandler_RMLP           : Variable        Mean        RMS   [        Min        Max ]
--- TFHandler_RMLP           : -----------------------------------------------------------
--- TFHandler_RMLP           :     var1:   0.085752    0.35735   [    -1.0000     1.0000 ]
--- TFHandler_RMLP           :     var2:    0.10321    0.36589   [    -1.0000     1.0000 ]
--- TFHandler_RMLP           :     var3:    0.10411    0.37276   [    -1.0000     1.0000 ]
--- TFHandler_RMLP           :     var4:    0.17623    0.40650   [    -1.0000     1.0000 ]
--- TFHandler_RMLP           : -----------------------------------------------------------
--- RMLP                     : Dataset[dataset] : Begin training
--- RMLP                     : 
--- RMLP                     : --- Saving State File In:weights/RMLPModel.RData
--- RMLP                     : 
--- RMLP                     : Dataset[dataset] : End of training                                              
--- RMLP                     : Dataset[dataset] : Elapsed time for training with 2000 events: 1.07 sec         
--- RMLP                     : Dataset[dataset] : Create MVA output for Dataset[dataset] : classification on training sample
--- RMLP                     : Dataset[dataset] : Evaluation of RMLP on training sample (2000 events)
--- RMLP                     : Dataset[dataset] : Elapsed time for evaluation of 2000 events: 0.739 sec       
--- RMLP                     : Dataset[dataset] : Creating weight file in xml format: weights/TMVAClassification_RMLP.weights.xml
--- RMLP                     : Dataset[dataset] : Creating standalone response class: weights/TMVAClassification_RMLP.class.C
--- Factory                  : Training finished
--- Factory                  : Train method: RXGB for Classification
--- RXGB                     : Dataset[dataset] : Begin training
[0]	train-rmse:0.393636
[1]	train-rmse:0.383795
[2]	train-rmse:0.371803
[3]	train-rmse:0.362196
[4]	train-rmse:0.357333
[5]	train-rmse:0.351344
[6]	train-rmse:0.346084
[7]	train-rmse:0.340683
[8]	train-rmse:0.332887
[9]	train-rmse:0.328419
[10]	train-rmse:0.326714
[11]	train-rmse:0.324244
[12]	train-rmse:0.321295
[13]	train-rmse:0.318570
[14]	train-rmse:0.315405
[15]	train-rmse:0.312603
[16]	train-rmse:0.310186
[17]	train-rmse:0.309424
[18]	train-rmse:0.306099
[19]	train-rmse:0.304363
--- RXGB                     : 
--- RXGB                     : --- Saving State File In:weights/RXGBModel.RData
--- RXGB                     : 
--- RXGB                     : Dataset[dataset] : End of training                                              
--- RXGB                     : Dataset[dataset] : Elapsed time for training with 2000 events: 0.234 sec         
--- RXGB                     : Dataset[dataset] : Create MVA output for Dataset[dataset] : classification on training sample
--- RXGB                     : Dataset[dataset] : Evaluation of RXGB on training sample (2000 events)
--- RXGB                     : Dataset[dataset] : Elapsed time for evaluation of 2000 events: 0.00485 sec       
--- RXGB                     : Dataset[dataset] : Creating weight file in xml format: weights/TMVAClassification_RXGB.weights.xml
--- Factory                  : Training finished
--- Factory                  : Train method: BDT for Classification
--- BDT                      : Dataset[dataset] : Begin training
--- BDT                      :  found and suggest the following possible pre-selection cuts 
--- BDT                      : as option DoPreselection was not used, these cuts however will not be performed, but the training will see the full sample
--- BDT                      :  found cut: Bkg if var 0 < -2.99038
--- BDT                      :  found cut: Bkg if var 2 < -2.88493
--- BDT                      :  found cut: Bkg if var 3 < -2.54088
--- BDT                      : <InitEventSample> For classification trees, 
--- BDT                      :  the effective number of backgrounds is scaled to match 
--- BDT                      :  the signal. Othersise the first boosting step would do 'just that'!
--- BDT                      : re-normlise events such that Sig and Bkg have respective sum of weights = 1
--- BDT                      :   sig->sig*1ev. bkg->bkg*1ev.
--- BDT                      : #events: (reweighted) sig: 1000 bkg: 1000
--- BDT                      : #events: (unweighted) sig: 1000 bkg: 1000
--- BDT                      : Training 50 Decision Trees ... patience please
--- BinaryTree               : The minimal node size MinNodeSize=2.5 fMinNodeSize=2.5% is translated to an actual number of events = 25.7 for the training sample size of 1028
--- BinaryTree               : Note: This number will be taken as absolute minimum in the node, 
--- BinaryTree               :       in terms of 'weighted events' and unweighted ones !! 
--- BDT                      : <Train> elapsed time: 0.0508 sec                              
--- BDT                      : <Train> average number of nodes (w/o pruning) : 4
--- BDT                      : Dataset[dataset] : End of training                                              
--- BDT                      : Dataset[dataset] : Elapsed time for training with 2000 events: 0.061 sec         
--- BDT                      : Dataset[dataset] : Create MVA output for Dataset[dataset] : classification on training sample
--- BDT                      : Dataset[dataset] : Evaluation of BDT on training sample (2000 events)
--- BDT                      : Dataset[dataset] : Elapsed time for evaluation of 2000 events: 0.00845 sec       
--- BDT                      : Dataset[dataset] : Creating weight file in xml format: dataset/weights/TMVAClassification_BDT.weights.xml
--- BDT                      : Dataset[dataset] : Creating standalone response class: dataset/weights/TMVAClassification_BDT.class.C
--- BDT                      : Write monitoring histograms to file: TMVAOutputCV.root:/dataset/Method_BDT/BDT
--- Factory                  : Training finished
--- Factory                  : 
--- Factory                  : Ranking input variables (method specific)...
--- Factory                  : No variable ranking supplied by classifier: C50
--- Factory                  : No variable ranking supplied by classifier: RMLP
--- Factory                  : No variable ranking supplied by classifier: RXGB
--- BDT                      : Ranking result (top variable is best ranked)
--- BDT                      : --------------------------------------
--- BDT                      : Rank : Variable  : Variable Importance
--- BDT                      : --------------------------------------
--- BDT                      :    1 : var4      : 4.608e-01
--- BDT                      :    2 : var1      : 2.923e-01
--- BDT                      :    3 : var2      : 1.457e-01
--- BDT                      :    4 : var3      : 1.012e-01
--- BDT                      : --------------------------------------
--- Factory                  : 
--- Factory                  : === Destroy and recreate all methods via weight files for testing ===
--- Factory                  : 
--- MethodBase               : Dataset[dataset] : Reading weight file: weights/TMVAClassification_C50.weights.xml
--- C50                      : Dataset[dataset] : Read method "C50" of type "C50"
--- C50                      : Dataset[dataset] : MVA method was trained with TMVA Version: 4.2.1
--- C50                      : Dataset[dataset] : MVA method was trained with ROOT Version: 6.07/07
--- MethodBase               : Dataset[dataset] : Reading weight file: weights/TMVAClassification_RMLP.weights.xml
--- RMLP                     : Dataset[dataset] : Read method "RMLP" of type "RSNNS"
--- RMLP                     : Dataset[dataset] : MVA method was trained with TMVA Version: 4.2.1
--- RMLP                     : Dataset[dataset] : MVA method was trained with ROOT Version: 6.07/07
--- MethodBase               : Dataset[dataset] : Reading weight file: weights/TMVAClassification_RXGB.weights.xml
--- RXGB                     : Dataset[dataset] : Read method "RXGB" of type "RXGB"
--- RXGB                     : Dataset[dataset] : MVA method was trained with TMVA Version: 4.2.1
--- RXGB                     : Dataset[dataset] : MVA method was trained with ROOT Version: 6.07/07
--- MethodBase               : Dataset[dataset] : Reading weight file: dataset/weights/TMVAClassification_BDT.weights.xml
--- BDT                      : Dataset[dataset] : Read method "BDT" of type "BDT"
--- BDT                      : Dataset[dataset] : MVA method was trained with TMVA Version: 4.2.1
--- BDT                      : Dataset[dataset] : MVA method was trained with ROOT Version: 6.07/07

Testing and Evaluating the data

In [7]:
factory.TestAllMethods();
factory.EvaluateAllMethods();
--- Factory                  : Test all methods...
--- Factory                  : Test method: C50 for Classification performance
--- C50                      : Dataset[dataset] : Evaluation of C50 on testing sample (10000 events)
--- C50                      : 
--- C50                      : --- Loading State File From:weights/C50Model.RData
--- C50                      : 
--- C50                      : Dataset[dataset] : Elapsed time for evaluation of 10000 events: 0.546 sec       
--- Factory                  : Test method: RMLP for Classification performance
--- RMLP                     : Dataset[dataset] : Evaluation of RMLP on testing sample (10000 events)
--- RMLP                     : 
--- RMLP                     : --- Loading State File From:weights/RMLPModel.RData
--- RMLP                     : 
--- RMLP                     : Dataset[dataset] : Elapsed time for evaluation of 10000 events: 3.49 sec       
--- Factory                  : Test method: RXGB for Classification performance
--- RXGB                     : Dataset[dataset] : Evaluation of RXGB on testing sample (10000 events)
--- RXGB                     : 
--- RXGB                     : --- Loading State File From:weights/RXGBModel.RData
--- RXGB                     : 
--- RXGB                     : Dataset[dataset] : Elapsed time for evaluation of 10000 events: 0.0286 sec       
--- Factory                  : Test method: BDT for Classification performance
--- BDT                      : Dataset[dataset] : Evaluation of BDT on testing sample (10000 events)
--- BDT                      : Dataset[dataset] : Elapsed time for evaluation of 10000 events: 0.0235 sec       
--- Factory                  : Evaluate all methods...
--- Factory                  : Evaluate classifier: C50
--- C50                      : Testing Classification C50 METHOD  
--- C50                      : Dataset[dataset] : Loop over test events and fill histograms with classifier response...
--- Factory                  : Write evaluation histograms to file
--- TFHandler_C50            : Plot event variables for C50
--- TFHandler_C50            : -----------------------------------------------------------
--- TFHandler_C50            : Variable        Mean        RMS   [        Min        Max ]
--- TFHandler_C50            : -----------------------------------------------------------
--- TFHandler_C50            :     var1: 0.00077102     1.6695   [    -5.8991     4.7639 ]
--- TFHandler_C50            :     var2: -0.0063164     1.5765   [    -5.2454     4.8300 ]
--- TFHandler_C50            :     var3:  -0.010870     1.7365   [    -5.3563     4.6430 ]
--- TFHandler_C50            :     var4:    0.14557     2.1608   [    -6.9675     5.0307 ]
--- TFHandler_C50            : -----------------------------------------------------------
--- TFHandler_C50            : Create scatter and profile plots in target-file directory: 
--- TFHandler_C50            : TMVAOutputCV.root:/dataset/Method_C50/C50/CorrelationPlots
--- Factory                  : Evaluate classifier: RMLP
--- RMLP                     : Testing Classification RMLP METHOD  
--- TFHandler_RMLP           : -----------------------------------------------------------
--- TFHandler_RMLP           : Variable        Mean        RMS   [        Min        Max ]
--- TFHandler_RMLP           : -----------------------------------------------------------
--- TFHandler_RMLP           :     var1:   0.066774    0.35913   [    -1.2024     1.0914 ]
--- TFHandler_RMLP           :     var2:   0.079492    0.36669   [    -1.1391     1.2044 ]
--- TFHandler_RMLP           :     var3:   0.079125    0.37282   [    -1.0685     1.0783 ]
--- TFHandler_RMLP           :     var4:    0.15120    0.40805   [    -1.1921     1.0737 ]
--- TFHandler_RMLP           : -----------------------------------------------------------
--- RMLP                     : Dataset[dataset] : Loop over test events and fill histograms with classifier response...
--- Factory                  : Write evaluation histograms to file
--- TFHandler_RMLP           : Plot event variables for RMLP
--- TFHandler_RMLP           : -----------------------------------------------------------
--- TFHandler_RMLP           : Variable        Mean        RMS   [        Min        Max ]
--- TFHandler_RMLP           : -----------------------------------------------------------
--- TFHandler_RMLP           :     var1:   0.066774    0.35913   [    -1.2024     1.0914 ]
--- TFHandler_RMLP           :     var2:   0.079492    0.36669   [    -1.1391     1.2044 ]
--- TFHandler_RMLP           :     var3:   0.079125    0.37282   [    -1.0685     1.0783 ]
--- TFHandler_RMLP           :     var4:    0.15120    0.40805   [    -1.1921     1.0737 ]
--- TFHandler_RMLP           : -----------------------------------------------------------
--- TFHandler_RMLP           : Create scatter and profile plots in target-file directory: 
--- TFHandler_RMLP           : TMVAOutputCV.root:/dataset/Method_RMLP/RMLP/CorrelationPlots
--- Factory                  : Evaluate classifier: RXGB
--- RXGB                     : Testing Classification RXGB METHOD  
--- RXGB                     : Dataset[dataset] : Loop over test events and fill histograms with classifier response...
--- Factory                  : Write evaluation histograms to file
--- TFHandler_RXGB           : Plot event variables for RXGB
--- TFHandler_RXGB           : -----------------------------------------------------------
--- TFHandler_RXGB           : Variable        Mean        RMS   [        Min        Max ]
--- TFHandler_RXGB           : -----------------------------------------------------------
--- TFHandler_RXGB           :     var1: 0.00077102     1.6695   [    -5.8991     4.7639 ]
--- TFHandler_RXGB           :     var2: -0.0063164     1.5765   [    -5.2454     4.8300 ]
--- TFHandler_RXGB           :     var3:  -0.010870     1.7365   [    -5.3563     4.6430 ]
--- TFHandler_RXGB           :     var4:    0.14557     2.1608   [    -6.9675     5.0307 ]
--- TFHandler_RXGB           : -----------------------------------------------------------
--- TFHandler_RXGB           : Create scatter and profile plots in target-file directory: 
--- TFHandler_RXGB           : TMVAOutputCV.root:/dataset/Method_RXGB/RXGB/CorrelationPlots
--- Factory                  : Evaluate classifier: BDT
--- BDT                      : Dataset[dataset] : Loop over test events and fill histograms with classifier response...
--- Factory                  : Write evaluation histograms to file
--- TFHandler_BDT            : Plot event variables for BDT
--- TFHandler_BDT            : -----------------------------------------------------------
--- TFHandler_BDT            : Variable        Mean        RMS   [        Min        Max ]
--- TFHandler_BDT            : -----------------------------------------------------------
--- TFHandler_BDT            :     var1: 0.00077102     1.6695   [    -5.8991     4.7639 ]
--- TFHandler_BDT            :     var2: -0.0063164     1.5765   [    -5.2454     4.8300 ]
--- TFHandler_BDT            :     var3:  -0.010870     1.7365   [    -5.3563     4.6430 ]
--- TFHandler_BDT            :     var4:    0.14557     2.1608   [    -6.9675     5.0307 ]
--- TFHandler_BDT            : -----------------------------------------------------------
--- TFHandler_BDT            : Create scatter and profile plots in target-file directory: 
--- TFHandler_BDT            : TMVAOutputCV.root:/dataset/Method_BDT/BDT/CorrelationPlots
--- Factory                  : 
--- Factory                  : Evaluation results ranked by best signal efficiency and purity (area)
--- Factory                  : -------------------------------------------------------------------------------------------------------------------
--- Factory                  : DataSet              MVA              Signal efficiency at bkg eff.(error):                | Sepa-    Signifi- 
--- Factory                  : Name:                Method:          @B=0.01    @B=0.10    @B=0.30    ROC-integ    ROCCurve| ration:  cance:   
--- Factory                  : -------------------------------------------------------------------------------------------------------------------
--- Factory                  : dataset              RMLP           : 0.343(06)  0.764(06)  0.954(02)    0.930       0.929 | 0.578    1.647
--- Factory                  : dataset              RXGB           : 0.208(05)  0.693(06)  0.921(03)    0.901       0.902 | 0.507    1.387
--- Factory                  : dataset              BDT            : 0.263(06)  0.642(06)  0.900(04)    0.894       0.894 | 0.483    1.248
--- Factory                  : dataset              C50            : 0.000(00)  0.689(06)  0.926(03)    0.892       0.898 | 0.521    1.411
--- Factory                  : -------------------------------------------------------------------------------------------------------------------
--- Factory                  : 
--- Factory                  : Testing efficiency compared to training efficiency (overtraining check)
--- Factory                  : -------------------------------------------------------------------------------------------------------------------
--- Factory                  : DataSet              MVA              Signal efficiency: from test sample (from training sample) 
--- Factory                  : Name:                Method:          @B=0.01             @B=0.10            @B=0.30   
--- Factory                  : -------------------------------------------------------------------------------------------------------------------
--- Factory                  : dataset              RMLP           : 0.343 (0.365)       0.764 (0.784)      0.954 (0.955)
--- Factory                  : dataset              RXGB           : 0.208 (0.375)       0.693 (0.802)      0.921 (0.944)
--- Factory                  : dataset              BDT            : 0.263 (0.225)       0.642 (0.671)      0.900 (0.902)
--- Factory                  : dataset              C50            : 0.000 (0.474)       0.689 (0.848)      0.926 (0.943)
--- Factory                  : -------------------------------------------------------------------------------------------------------------------
--- Factory                  : 
--- Dataset:dataset          : Dataset[dataset] : Created tree 'TestTree' with 10000 events
--- Dataset:dataset          : Dataset[dataset] : Created tree 'TrainTree' with 2000 events
--- Factory                  :   
--- Factory                  : Thank you for using TMVA!
--- Factory                  : For citation information, please visit: http://tmva.sf.net/citeTMVA.html

Ploting ROC Curve

We enable the ROOT JavaScript interactive visualisation.

In [8]:
%jsroot on
auto c = factory.GetROCCurve(&loader);
c->Draw();