Classification example of TMVA based on public Higgs UCI dataset
The UCI data set is a public HIGGS data set , see http://archive.ics.uci.edu/ml/datasets/HIGGS used in this paper: Baldi, P., P. Sadowski, and D. Whiteson. “Searching for Exotic Particles in High-energy Physics with Deep Learning.” Nature Communications 5 (July 2, 2014).
Author: Harshal Shende
This notebook tutorial was automatically generated with ROOTBOOK-izer from the macro found in the ROOT repository on Tuesday, March 19, 2024 at 07:20 PM.
Declare Factory
Create the Factory class. Later you can choose the methods whose performance you'd like to investigate.
The factory is the major TMVA object you have to interact with. Here is the list of parameters you need to pass
weightfiles in the directory weight/ that will be created with the method parameters
import ROOT
import os
TMVA = ROOT.TMVA
TFile = ROOT.TFile
TMVA.Tools.Instance()
<cppyy.gbl.TMVA.Tools object at 0x8e00130>
options to control used methods
useLikelihood = True # likelihood based discriminant
useLikelihoodKDE = False # likelihood based discriminant
useFischer = True # Fischer discriminant
useMLP = False # Multi Layer Perceptron (old TMVA NN implementation)
useBDT = True # Boosted Decision Tree
useDL = True # TMVA Deep learning ( CPU or GPU)
useKeras = True # Use Keras Deep Learning via PyMVA
if ROOT.gSystem.GetFromPipe("root-config --has-tmva-pymva") == "yes":
TMVA.PyMethodBase.PyInitialize()
else:
useKeras = False # cannot use Keras if PYMVA is not available
if useKeras:
try:
import tensorflow
except:
ROOT.Warning("TMVA_Higgs_Classification", "Skip using Keras since tensorflow is not available")
useKeras = False
outputFile = TFile.Open("Higgs_ClassificationOutput.root", "RECREATE")
factory = TMVA.Factory(
"TMVA_Higgs_Classification", outputFile, V=False, ROC=True, Silent=False, Color=True, AnalysisType="Classification"
)
2024-03-19 19:20:28.956398: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used. 2024-03-19 19:20:29.034698: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used. 2024-03-19 19:20:29.040637: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-03-19 19:20:31.416693: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Setup Dataset(s)
Define now input data file and signal and background trees
inputFileName = "Higgs_data.root"
inputFileLink = "http://root.cern.ch/files/" + inputFileName
if ROOT.gSystem.AccessPathName(inputFileName):
ROOT.Info("TMVA_Higgs_Classification", "Download Higgs_data.root file")
TFile.SetCacheFileDir(".")
inputFile = TFile.Open(inputFileLink, "CACHEREAD")
if inputFile is None:
raise FileNotFoundError("Input file cannot be downloaded - exit")
else:
# file exists
inputFile = TFile.Open(inputFileName)
Info in <TMVA_Higgs_Classification>: Download Higgs_data.root file [TFile::Cp] Total 1.50 MB |====================| 100.00 % [13.0 MB/s] Info in <TFile::OpenFromCache>: using local cache copy of http://root.cern.ch/files/Higgs_data.root [./files/Higgs_data.root]
--- Register the training and test trees
signalTree = inputFile.Get("sig_tree")
backgroundTree = inputFile.Get("bkg_tree")
signalTree.Print()
****************************************************************************** *Tree :sig_tree : tree * *Entries : 10000 : Total = 1177229 bytes File Size = 785298 * * : : Tree compression factor = 1.48 * ****************************************************************************** *Br 0 :Type : Type/F * *Entries : 10000 : Total Size= 40556 bytes File Size = 307 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 130.54 * *............................................................................* *Br 1 :lepton_pT : lepton_pT/F * *Entries : 10000 : Total Size= 40581 bytes File Size = 30464 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.32 * *............................................................................* *Br 2 :lepton_eta : lepton_eta/F * *Entries : 10000 : Total Size= 40586 bytes File Size = 28650 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.40 * *............................................................................* *Br 3 :lepton_phi : lepton_phi/F * *Entries : 10000 : Total Size= 40586 bytes File Size = 30508 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.31 * *............................................................................* *Br 4 :missing_energy_magnitude : missing_energy_magnitude/F * *Entries : 10000 : Total Size= 40656 bytes File Size = 35749 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.12 * *............................................................................* *Br 5 :missing_energy_phi : missing_energy_phi/F * *Entries : 10000 : Total Size= 40626 bytes File Size = 36766 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.09 * *............................................................................* *Br 6 :jet1_pt : jet1_pt/F * *Entries : 10000 : Total Size= 40571 bytes File Size = 32298 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.24 * *............................................................................* *Br 7 :jet1_eta : jet1_eta/F * *Entries : 10000 : Total Size= 40576 bytes File Size = 28467 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.41 * *............................................................................* *Br 8 :jet1_phi : jet1_phi/F * *Entries : 10000 : Total Size= 40576 bytes File Size = 30399 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.32 * *............................................................................* *Br 9 :jet1_b-tag : jet1_b-tag/F * *Entries : 10000 : Total Size= 40586 bytes File Size = 5087 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 7.88 * *............................................................................* *Br 10 :jet2_pt : jet2_pt/F * *Entries : 10000 : Total Size= 40571 bytes File Size = 31561 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.27 * *............................................................................* *Br 11 :jet2_eta : jet2_eta/F * *Entries : 10000 : Total Size= 40576 bytes File Size = 28616 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.40 * *............................................................................* *Br 12 :jet2_phi : jet2_phi/F * *Entries : 10000 : Total Size= 40576 bytes File Size = 30547 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.31 * *............................................................................* *Br 13 :jet2_b-tag : jet2_b-tag/F * *Entries : 10000 : Total Size= 40586 bytes File Size = 5031 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 7.97 * *............................................................................* *Br 14 :jet3_pt : jet3_pt/F * *Entries : 10000 : Total Size= 40571 bytes File Size = 30642 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.31 * *............................................................................* *Br 15 :jet3_eta : jet3_eta/F * *Entries : 10000 : Total Size= 40576 bytes File Size = 28955 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.38 * *............................................................................* *Br 16 :jet3_phi : jet3_phi/F * *Entries : 10000 : Total Size= 40576 bytes File Size = 30433 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.32 * *............................................................................* *Br 17 :jet3_b-tag : jet3_b-tag/F * *Entries : 10000 : Total Size= 40586 bytes File Size = 4879 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 8.22 * *............................................................................* *Br 18 :jet4_pt : jet4_pt/F * *Entries : 10000 : Total Size= 40571 bytes File Size = 29189 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.37 * *............................................................................* *Br 19 :jet4_eta : jet4_eta/F * *Entries : 10000 : Total Size= 40576 bytes File Size = 29311 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.37 * *............................................................................* *Br 20 :jet4_phi : jet4_phi/F * *Entries : 10000 : Total Size= 40576 bytes File Size = 30525 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.31 * *............................................................................* *Br 21 :jet4_b-tag : jet4_b-tag/F * *Entries : 10000 : Total Size= 40586 bytes File Size = 4725 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 8.48 * *............................................................................* *Br 22 :m_jj : m_jj/F * *Entries : 10000 : Total Size= 40556 bytes File Size = 34991 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.15 * *............................................................................* *Br 23 :m_jjj : m_jjj/F * *Entries : 10000 : Total Size= 40561 bytes File Size = 34460 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.16 * *............................................................................* *Br 24 :m_lv : m_lv/F * *Entries : 10000 : Total Size= 40556 bytes File Size = 32232 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.24 * *............................................................................* *Br 25 :m_jlv : m_jlv/F * *Entries : 10000 : Total Size= 40561 bytes File Size = 34598 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.16 * *............................................................................* *Br 26 :m_bb : m_bb/F * *Entries : 10000 : Total Size= 40556 bytes File Size = 35012 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.14 * *............................................................................* *Br 27 :m_wbb : m_wbb/F * *Entries : 10000 : Total Size= 40561 bytes File Size = 34493 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.16 * *............................................................................* *Br 28 :m_wwbb : m_wwbb/F * *Entries : 10000 : Total Size= 40566 bytes File Size = 34410 * *Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.16 * *............................................................................*
Declare DataLoader(s)
The next step is to declare the DataLoader class that deals with input variables Define the input variables that shall be used for the MVA training note that you may also use variable expressions, which can be parsed by TTree::Draw( "expression" )]
loader = TMVA.DataLoader("dataset")
loader.AddVariable("m_jj")
loader.AddVariable("m_jjj")
loader.AddVariable("m_lv")
loader.AddVariable("m_jlv")
loader.AddVariable("m_bb")
loader.AddVariable("m_wbb")
loader.AddVariable("m_wwbb")
We set now the input data trees in the TMVA DataLoader class global event weights per tree (see below for setting event-wise weights)
signalWeight = 1.0
backgroundWeight = 1.0
You can add an arbitrary number of signal or background trees
loader.AddSignalTree(signalTree, signalWeight)
loader.AddBackgroundTree(backgroundTree, backgroundWeight)
DataSetInfo : [dataset] : Added class "Signal" : Add Tree sig_tree of type Signal with 10000 events DataSetInfo : [dataset] : Added class "Background" : Add Tree bkg_tree of type Background with 10000 events
Set individual event weights (the variables must exist in the original TTree) for signal : factory->SetSignalWeightExpression ("weight1weight2"); for background: factory->SetBackgroundWeightExpression("weight1weight2"); loader->SetBackgroundWeightExpression( "weight" );
Apply additional cuts on the signal and background samples (can be different)
mycuts = ROOT.TCut("") # for example: TCut mycuts = "abs(var1)<0.5 && abs(var2-0.5)<1";
mycutb = ROOT.TCut("") # for example: TCut mycutb = "abs(var1)<0.5";
Tell the factory how to use the training and testing events
If no numbers of events are given, half of the events in the tree are used for training, and the other half for testing: loader->PrepareTrainingAndTestTree( mycut, "SplitMode=random:!V" ); To also specify the number of testing events, use:
loader.PrepareTrainingAndTestTree(
mycuts, mycutb, nTrain_Signal=7000, nTrain_Background=7000, SplitMode="Random", NormMode="NumEvents", V=False
)
Booking Methods
Here we book the TMVA methods. We book first a Likelihood based on KDE (Kernel Density Estimation), a Fischer discriminant, a BDT and a shallow neural network Likelihood ("naive Bayes estimator")
if useLikelihood:
factory.BookMethod(
loader,
TMVA.Types.kLikelihood,
"Likelihood",
H=True,
V=False,
TransformOutput=True,
PDFInterpol="Spline2:NSmoothSig[0]=20:NSmoothBkg[0]=20:NSmoothBkg[1]=10",
NSmooth=1,
NAvEvtPerBin=50,
)
Factory : Booking method: Likelihood
:
Use a kernel density estimator to approximate the PDFs
if useLikelihoodKDE:
factory.BookMethod(
loader,
TMVA.Types.kLikelihood,
"LikelihoodKDE",
H=False,
V=False,
TransformOutput=False,
PDFInterpol="KDE",
KDEtype="Gauss",
KDEiter="Adaptive",
KDEFineFactor=0.3,
KDEborder=None,
NAvEvtPerBin=50,
)
Fisher discriminant (same as LD)
if useFischer:
factory.BookMethod(
loader,
TMVA.Types.kFisher,
"Fisher",
H=True,
V=False,
Fisher=True,
VarTransform=None,
CreateMVAPdfs=True,
PDFInterpolMVAPdf="Spline2",
NbinsMVAPdf=50,
NsmoothMVAPdf=10,
)
Factory : Booking method: Fisher
:
Boosted Decision Trees
if useBDT:
factory.BookMethod(
loader,
TMVA.Types.kBDT,
"BDT",
V=False,
NTrees=200,
MinNodeSize="2.5%",
MaxDepth=2,
BoostType="AdaBoost",
AdaBoostBeta=0.5,
UseBaggedBoost=True,
BaggedSampleFraction=0.5,
SeparationType="GiniIndex",
nCuts=20,
)
Factory : Booking method: BDT
:
: Rebuilding Dataset dataset
: Building event vectors for type 2 Signal
: Dataset[dataset] : create input formulas for tree sig_tree
: Building event vectors for type 2 Background
: Dataset[dataset] : create input formulas for tree bkg_tree
DataSetFactory : [dataset] : Number of events in input trees
:
:
: Number of training and testing events
: ---------------------------------------------------------------------------
: Signal -- training events : 7000
: Signal -- testing events : 3000
: Signal -- training and testing events: 10000
: Background -- training events : 7000
: Background -- testing events : 3000
: Background -- training and testing events: 10000
:
DataSetInfo : Correlation matrix (Signal):
: ----------------------------------------------------------------
: m_jj m_jjj m_lv m_jlv m_bb m_wbb m_wwbb
: m_jj: +1.000 +0.774 -0.004 +0.096 +0.024 +0.512 +0.533
: m_jjj: +0.774 +1.000 -0.010 +0.073 +0.152 +0.674 +0.668
: m_lv: -0.004 -0.010 +1.000 +0.121 -0.027 +0.009 +0.021
: m_jlv: +0.096 +0.073 +0.121 +1.000 +0.313 +0.544 +0.552
: m_bb: +0.024 +0.152 -0.027 +0.313 +1.000 +0.445 +0.333
: m_wbb: +0.512 +0.674 +0.009 +0.544 +0.445 +1.000 +0.915
: m_wwbb: +0.533 +0.668 +0.021 +0.552 +0.333 +0.915 +1.000
: ----------------------------------------------------------------
DataSetInfo : Correlation matrix (Background):
: ----------------------------------------------------------------
: m_jj m_jjj m_lv m_jlv m_bb m_wbb m_wwbb
: m_jj: +1.000 +0.808 +0.022 +0.150 +0.028 +0.407 +0.415
: m_jjj: +0.808 +1.000 +0.041 +0.206 +0.177 +0.569 +0.547
: m_lv: +0.022 +0.041 +1.000 +0.139 +0.037 +0.081 +0.085
: m_jlv: +0.150 +0.206 +0.139 +1.000 +0.309 +0.607 +0.557
: m_bb: +0.028 +0.177 +0.037 +0.309 +1.000 +0.625 +0.447
: m_wbb: +0.407 +0.569 +0.081 +0.607 +0.625 +1.000 +0.884
: m_wwbb: +0.415 +0.547 +0.085 +0.557 +0.447 +0.884 +1.000
: ----------------------------------------------------------------
DataSetFactory : [dataset] :
:
Multi-Layer Perceptron (Neural Network)
if useMLP:
factory.BookMethod(
loader,
TMVA.Types.kMLP,
"MLP",
H=False,
V=False,
NeuronType="tanh",
VarTransform="N",
NCycles=100,
HiddenLayers="N+5",
TestRate=5,
UseRegulator=False,
)
Here we book the new DNN of TMVA if we have support in ROOT. We will use GPU version if ROOT is enabled with GPU
Booking Deep Neural Network
Here we define the option string for building the Deep Neural network model.
The DNN configuration is defined using a string. Note that whitespaces between characters are not allowed.
We define first the DNN layout:
input depth | height | width
.In case of a dense layer as first layer the input layout should be 1 | 1 | number of input variables
(features)
If the first layer is dense it should be 1 | batch size ! number of variables
(features)
(note the use of the character |
as separator of input parameters for DNN layout)
note that in case of only dense layer the input layout could be omitted but it is required when defining more complex architectures
the different layers are separated by the ","
We define here the training strategy parameters for the DNN. The parameters are separated by the ","
separator.
One can then concatenate different training strategy with different parameters. The training strategy are separated by
the "|"
separator.
We define the general DNN options concatenating in the final string the previously defined layout and training strategy.
Note we use the ":"
separator to separate the different higher level options, as in the other TMVA methods.
In addition to input layout, batch layout and training strategy we add now:
We can then book the DL method using the built option string
if useDL:
useDLGPU = ROOT.gSystem.GetFromPipe("root-config --has-tmva-gpu") == "yes"
# Define DNN layout
# Define Training strategies
# one can catenate several training strategies
training1 = ROOT.TString(
"LearningRate=1e-3,Momentum=0.9,"
"ConvergenceSteps=10,BatchSize=128,TestRepetitions=1,"
"MaxEpochs=20,WeightDecay=1e-4,Regularization=None,"
"Optimizer=ADAM,ADAM_beta1=0.9,ADAM_beta2=0.999,ADAM_eps=1.E-7," # ADAM default parameters
"DropConfig=0.0+0.0+0.0+0."
)
# training2 = ROOT.TString("LearningRate=1e-3,Momentum=0.9"
# "ConvergenceSteps=10,BatchSize=128,TestRepetitions=1,"
# "MaxEpochs=20,WeightDecay=1e-4,Regularization=None,"
# "Optimizer=SGD,DropConfig=0.0+0.0+0.0+0.")
# General Options.
dnnMethodName = ROOT.TString("DNN_CPU")
if useDLGPU:
arch = "GPU"
dnnMethodName = "DNN_GPU"
else:
arch = "CPU"
factory.BookMethod(
loader,
TMVA.Types.kDL,
dnnMethodName,
H=False,
V=True,
ErrorStrategy="CROSSENTROPY",
VarTransform="G",
WeightInitialization="XAVIER",
InputLayout="1|1|7",
BatchLayout="1|128|7",
Layout="DENSE|64|TANH,DENSE|64|TANH,DENSE|64|TANH,DENSE|64|TANH,DENSE|1|LINEAR",
TrainingStrategy=training1,
Architecture=arch,
)
Factory : Booking method: DNN_CPU
:
: Parsing option string:
: ... "!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=G:WeightInitialization=XAVIER:InputLayout=1|1|7:BatchLayout=1|128|7:Layout=DENSE|64|TANH,DENSE|64|TANH,DENSE|64|TANH,DENSE|64|TANH,DENSE|1|LINEAR:TrainingStrategy=LearningRate=1e-3,Momentum=0.9,ConvergenceSteps=10,BatchSize=128,TestRepetitions=1,MaxEpochs=20,WeightDecay=1e-4,Regularization=None,Optimizer=ADAM,ADAM_beta1=0.9,ADAM_beta2=0.999,ADAM_eps=1.E-7,DropConfig=0.0+0.0+0.0+0.:Architecture=CPU"
: The following options are set:
: - By User:
: <none>
: - Default:
: Boost_num: "0" [Number of times the classifier will be boosted]
: Parsing option string:
: ... "!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=G:WeightInitialization=XAVIER:InputLayout=1|1|7:BatchLayout=1|128|7:Layout=DENSE|64|TANH,DENSE|64|TANH,DENSE|64|TANH,DENSE|64|TANH,DENSE|1|LINEAR:TrainingStrategy=LearningRate=1e-3,Momentum=0.9,ConvergenceSteps=10,BatchSize=128,TestRepetitions=1,MaxEpochs=20,WeightDecay=1e-4,Regularization=None,Optimizer=ADAM,ADAM_beta1=0.9,ADAM_beta2=0.999,ADAM_eps=1.E-7,DropConfig=0.0+0.0+0.0+0.:Architecture=CPU"
: The following options are set:
: - By User:
: V: "True" [Verbose output (short form of "VerbosityLevel" below - overrides the latter one)]
: VarTransform: "G" [List of variable transformations performed before training, e.g., "D_Background,P_Signal,G,N_AllClasses" for: "Decorrelation, PCA-transformation, Gaussianisation, Normalisation, each for the given class of events ('AllClasses' denotes all events of all classes, if no class indication is given, 'All' is assumed)"]
: H: "False" [Print method-specific help message]
: InputLayout: "1|1|7" [The Layout of the input]
: BatchLayout: "1|128|7" [The Layout of the batch]
: Layout: "DENSE|64|TANH,DENSE|64|TANH,DENSE|64|TANH,DENSE|64|TANH,DENSE|1|LINEAR" [Layout of the network.]
: ErrorStrategy: "CROSSENTROPY" [Loss function: Mean squared error (regression) or cross entropy (binary classification).]
: WeightInitialization: "XAVIER" [Weight initialization strategy]
: Architecture: "CPU" [Which architecture to perform the training on.]
: TrainingStrategy: "LearningRate=1e-3,Momentum=0.9,ConvergenceSteps=10,BatchSize=128,TestRepetitions=1,MaxEpochs=20,WeightDecay=1e-4,Regularization=None,Optimizer=ADAM,ADAM_beta1=0.9,ADAM_beta2=0.999,ADAM_eps=1.E-7,DropConfig=0.0+0.0+0.0+0." [Defines the training strategies.]
: - Default:
: VerbosityLevel: "Default" [Verbosity level]
: CreateMVAPdfs: "False" [Create PDFs for classifier outputs (signal and background)]
: IgnoreNegWeightsInTraining: "False" [Events with negative weights are ignored in the training (but are included for testing and performance evaluation)]
: RandomSeed: "0" [Random seed used for weight initialization and batch shuffling]
: ValidationSize: "20%" [Part of the training data to use for validation. Specify as 0.2 or 20% to use a fifth of the data set as validation set. Specify as 100 to use exactly 100 events. (Default: 20%)]
DNN_CPU : [dataset] : Create Transformation "G" with events from all classes.
:
: Transformation, Variable selection :
: Input : variable 'm_jj' <---> Output : variable 'm_jj'
: Input : variable 'm_jjj' <---> Output : variable 'm_jjj'
: Input : variable 'm_lv' <---> Output : variable 'm_lv'
: Input : variable 'm_jlv' <---> Output : variable 'm_jlv'
: Input : variable 'm_bb' <---> Output : variable 'm_bb'
: Input : variable 'm_wbb' <---> Output : variable 'm_wbb'
: Input : variable 'm_wwbb' <---> Output : variable 'm_wwbb'
: Will now use the CPU architecture with BLAS and IMT support !
Keras DL
if useKeras:
ROOT.Info("TMVA_Higgs_Classification", "Building Deep Learning keras model")
# create Keras model with 4 layers of 64 units and relu activations
import tensorflow
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.layers import Input, Dense
model = Sequential()
model.add(Dense(64, activation="relu", input_dim=7))
model.add(Dense(64, activation="relu"))
model.add(Dense(64, activation="relu"))
model.add(Dense(64, activation="relu"))
model.add(Dense(2, activation="sigmoid"))
model.compile(loss="binary_crossentropy", optimizer=Adam(learning_rate=0.001), weighted_metrics=["accuracy"])
model.save("model_higgs.h5")
model.summary()
if not os.path.exists("model_higgs.h5"):
raise FileNotFoundError("Error creating Keras model file - skip using Keras")
else:
# book PyKeras method only if Keras model could be created
ROOT.Info("TMVA_Higgs_Classification", "Booking Deep Learning keras model")
factory.BookMethod(
loader,
TMVA.Types.kPyKeras,
"PyKeras",
H=True,
V=False,
VarTransform=None,
FilenameModel="model_higgs.h5",
FilenameTrainedModel="trained_model_higgs.h5",
NumEpochs=20,
BatchSize=100,
)
Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= dense (Dense) (None, 64) 512 dense_1 (Dense) (None, 64) 4160 dense_2 (Dense) (None, 64) 4160 dense_3 (Dense) (None, 64) 4160 dense_4 (Dense) (None, 2) 130 ================================================================= Total params: 13122 (51.26 KB) Trainable params: 13122 (51.26 KB) Non-trainable params: 0 (0.00 Byte) _________________________________________________________________
/usr/local/lib/python3.8/dist-packages/keras/src/engine/training.py:3000: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`. saving_api.save_model(
Factory : Booking method: PyKeras
:
: Setting up tf.keras
: Using TensorFlow version 2
: Use Keras version from TensorFlow : tf.keras
: Loading Keras Model
: Loaded model from file: model_higgs.h5
Info in <TMVA_Higgs_Classification>: Building Deep Learning keras model Info in <TMVA_Higgs_Classification>: Booking Deep Learning keras model
GpuOptions="allow_growth=True",
) # needed for RTX NVidia card and to avoid TF allocates all GPU memory
Train Methods
Here we train all the previously booked methods.
factory.TrainAllMethods()
Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= dense (Dense) (None, 64) 512 dense_1 (Dense) (None, 64) 4160 dense_2 (Dense) (None, 64) 4160 dense_3 (Dense) (None, 64) 4160 dense_4 (Dense) (None, 2) 130 ================================================================= Total params: 13122 (51.26 KB) Trainable params: 13122 (51.26 KB) Non-trainable params: 0 (0.00 Byte) _________________________________________________________________ Epoch 1/20 112/112 [==============================] - ETA: 0s - loss: 0.6729 - accuracy: 0.5752 Epoch 1: val_loss improved from inf to 0.65249, saving model to trained_model_higgs.h5 112/112 [==============================] - 4s 21ms/step - loss: 0.6729 - accuracy: 0.5752 - val_loss: 0.6525 - val_accuracy: 0.6250 Epoch 2/20 107/112 [===========================>..] - ETA: 0s - loss: 0.6389 - accuracy: 0.6349 Epoch 2: val_loss improved from 0.65249 to 0.62793, saving model to trained_model_higgs.h5 112/112 [==============================] - 1s 10ms/step - loss: 0.6387 - accuracy: 0.6346 - val_loss: 0.6279 - val_accuracy: 0.6600 Epoch 3/20 111/112 [============================>.] - ETA: 0s - loss: 0.6198 - accuracy: 0.6579 Epoch 3: val_loss improved from 0.62793 to 0.61984, saving model to trained_model_higgs.h5 112/112 [==============================] - 2s 14ms/step - loss: 0.6201 - accuracy: 0.6571 - val_loss: 0.6198 - val_accuracy: 0.6525 Epoch 4/20 110/112 [============================>.] - ETA: 0s - loss: 0.6124 - accuracy: 0.6642 Epoch 4: val_loss improved from 0.61984 to 0.61338, saving model to trained_model_higgs.h5 112/112 [==============================] - 1s 10ms/step - loss: 0.6123 - accuracy: 0.6642 - val_loss: 0.6134 - val_accuracy: 0.6611 Epoch 5/20 109/112 [============================>.] - ETA: 0s - loss: 0.6066 - accuracy: 0.6709 Epoch 5: val_loss improved from 0.61338 to 0.60365, saving model to trained_model_higgs.h5 112/112 [==============================] - 2s 15ms/step - loss: 0.6076 - accuracy: 0.6699 - val_loss: 0.6036 - val_accuracy: 0.6675 Epoch 6/20 111/112 [============================>.] - ETA: 0s - loss: 0.6023 - accuracy: 0.6752 Epoch 6: val_loss did not improve from 0.60365 112/112 [==============================] - 1s 13ms/step - loss: 0.6020 - accuracy: 0.6756 - val_loss: 0.6146 - val_accuracy: 0.6629 Epoch 7/20 112/112 [==============================] - ETA: 0s - loss: 0.6021 - accuracy: 0.6735 Epoch 7: val_loss improved from 0.60365 to 0.60174, saving model to trained_model_higgs.h5 112/112 [==============================] - 2s 18ms/step - loss: 0.6021 - accuracy: 0.6735 - val_loss: 0.6017 - val_accuracy: 0.6682 Epoch 8/20 107/112 [===========================>..] - ETA: 0s - loss: 0.5933 - accuracy: 0.6787 Epoch 8: val_loss improved from 0.60174 to 0.60021, saving model to trained_model_higgs.h5 112/112 [==============================] - 2s 18ms/step - loss: 0.5941 - accuracy: 0.6785 - val_loss: 0.6002 - val_accuracy: 0.6654 Epoch 9/20 112/112 [==============================] - ETA: 0s - loss: 0.5918 - accuracy: 0.6801 Epoch 9: val_loss improved from 0.60021 to 0.59633, saving model to trained_model_higgs.h5 112/112 [==============================] - 2s 15ms/step - loss: 0.5918 - accuracy: 0.6801 - val_loss: 0.5963 - val_accuracy: 0.6782 Epoch 10/20 112/112 [==============================] - ETA: 0s - loss: 0.5906 - accuracy: 0.6813 Epoch 10: val_loss did not improve from 0.59633 112/112 [==============================] - 1s 10ms/step - loss: 0.5906 - accuracy: 0.6813 - val_loss: 0.5985 - val_accuracy: 0.6643 Epoch 11/20 110/112 [============================>.] - ETA: 0s - loss: 0.5917 - accuracy: 0.6773 Epoch 11: val_loss did not improve from 0.59633 112/112 [==============================] - 1s 10ms/step - loss: 0.5914 - accuracy: 0.6779 - val_loss: 0.5984 - val_accuracy: 0.6729 Epoch 12/20 99/112 [=========================>....] - ETA: 0s - loss: 0.5868 - accuracy: 0.6827 Epoch 12: val_loss improved from 0.59633 to 0.59369, saving model to trained_model_higgs.h5 112/112 [==============================] - 1s 10ms/step - loss: 0.5847 - accuracy: 0.6840 - val_loss: 0.5937 - val_accuracy: 0.6757 Epoch 13/20 104/112 [==========================>...] - ETA: 0s - loss: 0.5827 - accuracy: 0.6881 Epoch 13: val_loss did not improve from 0.59369 112/112 [==============================] - 2s 13ms/step - loss: 0.5839 - accuracy: 0.6874 - val_loss: 0.5989 - val_accuracy: 0.6707 Epoch 14/20 106/112 [===========================>..] - ETA: 0s - loss: 0.5822 - accuracy: 0.6861 Epoch 14: val_loss improved from 0.59369 to 0.59337, saving model to trained_model_higgs.h5 112/112 [==============================] - 2s 20ms/step - loss: 0.5821 - accuracy: 0.6869 - val_loss: 0.5934 - val_accuracy: 0.6757 Epoch 15/20 112/112 [==============================] - ETA: 0s - loss: 0.5813 - accuracy: 0.6888 Epoch 15: val_loss improved from 0.59337 to 0.59001, saving model to trained_model_higgs.h5 112/112 [==============================] - 2s 15ms/step - loss: 0.5813 - accuracy: 0.6888 - val_loss: 0.5900 - val_accuracy: 0.6782 Epoch 16/20 107/112 [===========================>..] - ETA: 0s - loss: 0.5782 - accuracy: 0.6862 Epoch 16: val_loss did not improve from 0.59001 112/112 [==============================] - 2s 16ms/step - loss: 0.5776 - accuracy: 0.6864 - val_loss: 0.5908 - val_accuracy: 0.6718 Epoch 17/20 112/112 [==============================] - ETA: 0s - loss: 0.5775 - accuracy: 0.6903 Epoch 17: val_loss improved from 0.59001 to 0.58863, saving model to trained_model_higgs.h5 112/112 [==============================] - 2s 15ms/step - loss: 0.5775 - accuracy: 0.6903 - val_loss: 0.5886 - val_accuracy: 0.6782 Epoch 18/20 110/112 [============================>.] - ETA: 0s - loss: 0.5748 - accuracy: 0.6943 Epoch 18: val_loss did not improve from 0.58863 112/112 [==============================] - 1s 11ms/step - loss: 0.5751 - accuracy: 0.6940 - val_loss: 0.5906 - val_accuracy: 0.6750 Epoch 19/20 111/112 [============================>.] - ETA: 0s - loss: 0.5736 - accuracy: 0.6935 Epoch 19: val_loss improved from 0.58863 to 0.58778, saving model to trained_model_higgs.h5 112/112 [==============================] - 1s 13ms/step - loss: 0.5729 - accuracy: 0.6940 - val_loss: 0.5878 - val_accuracy: 0.6775 Epoch 20/20 112/112 [==============================] - ETA: 0s - loss: 0.5707 - accuracy: 0.6953 Epoch 20: val_loss did not improve from 0.58778 112/112 [==============================] - 1s 12ms/step - loss: 0.5707 - accuracy: 0.6953 - val_loss: 0.5893 - val_accuracy: 0.6743
/usr/local/lib/python3.8/dist-packages/keras/src/engine/training_v1.py:2359: UserWarning: `Model.state_updates` will be removed in a future version. This property should not be used in TensorFlow 2.0, as `updates` are applied automatically. updates=self.state_updates,
Factory : Train all methods Factory : [dataset] : Create Transformation "I" with events from all classes. : : Transformation, Variable selection : : Input : variable 'm_jj' <---> Output : variable 'm_jj' : Input : variable 'm_jjj' <---> Output : variable 'm_jjj' : Input : variable 'm_lv' <---> Output : variable 'm_lv' : Input : variable 'm_jlv' <---> Output : variable 'm_jlv' : Input : variable 'm_bb' <---> Output : variable 'm_bb' : Input : variable 'm_wbb' <---> Output : variable 'm_wbb' : Input : variable 'm_wwbb' <---> Output : variable 'm_wwbb' TFHandler_Factory : Variable Mean RMS [ Min Max ] : ----------------------------------------------------------- : m_jj: 1.0318 0.65629 [ 0.15106 16.132 ] : m_jjj: 1.0217 0.37420 [ 0.34247 8.9401 ] : m_lv: 1.0507 0.16678 [ 0.26679 3.6823 ] : m_jlv: 1.0161 0.40288 [ 0.38441 6.5831 ] : m_bb: 0.97707 0.53961 [ 0.080986 8.2551 ] : m_wbb: 1.0358 0.36856 [ 0.38503 6.4013 ] : m_wwbb: 0.96265 0.31608 [ 0.43228 4.5350 ] : ----------------------------------------------------------- : Ranking input variables (method unspecific)... IdTransformation : Ranking result (top variable is best ranked) : ------------------------------- : Rank : Variable : Separation : ------------------------------- : 1 : m_bb : 9.511e-02 : 2 : m_wbb : 4.268e-02 : 3 : m_wwbb : 4.178e-02 : 4 : m_jjj : 2.825e-02 : 5 : m_jlv : 1.999e-02 : 6 : m_jj : 3.834e-03 : 7 : m_lv : 3.699e-03 : ------------------------------- Factory : Train method: Likelihood for Classification : : : ================================================================ : H e l p f o r M V A m e t h o d [ Likelihood ] : : : --- Short description: : : The maximum-likelihood classifier models the data with probability : density functions (PDF) reproducing the signal and background : distributions of the input variables. Correlations among the : variables are ignored. : : --- Performance optimisation: : : Required for good performance are decorrelated input variables : (PCA transformation via the option "VarTransform=Decorrelate" : may be tried). Irreducible non-linear correlations may be reduced : by precombining strongly correlated input variables, or by simply : removing one of the variables. : : --- Performance tuning via configuration options: : : High fidelity PDF estimates are mandatory, i.e., sufficient training : statistics is required to populate the tails of the distributions : It would be a surprise if the default Spline or KDE kernel parameters : provide a satisfying fit to the data. The user is advised to properly : tune the events per bin and smooth options in the spline cases : individually per variable. If the KDE kernel is used, the adaptive : Gaussian kernel may lead to artefacts, so please always also try : the non-adaptive one. : : All tuning parameters must be adjusted individually for each input : variable! : : <Suppress this message by specifying "!H" in the booking option> : ================================================================ : : Filling reference histograms : Building PDF out of reference histograms : Elapsed time for training with 14000 events: 0.162 sec Likelihood : [dataset] : Evaluation of Likelihood on training sample (14000 events) : Elapsed time for evaluation of 14000 events: 0.0247 sec : Creating xml weight file: dataset/weights/TMVA_Higgs_Classification_Likelihood.weights.xml : Creating standalone class: dataset/weights/TMVA_Higgs_Classification_Likelihood.class.C : Higgs_ClassificationOutput.root:/dataset/Method_Likelihood/Likelihood Factory : Training finished : Factory : Train method: Fisher for Classification : : : ================================================================ : H e l p f o r M V A m e t h o d [ Fisher ] : : : --- Short description: : : Fisher discriminants select events by distinguishing the mean : values of the signal and background distributions in a trans- : formed variable space where linear correlations are removed. : : (More precisely: the "linear discriminator" determines : an axis in the (correlated) hyperspace of the input : variables such that, when projecting the output classes : (signal and background) upon this axis, they are pushed : as far as possible away from each other, while events : of a same class are confined in a close vicinity. The : linearity property of this classifier is reflected in the : metric with which "far apart" and "close vicinity" are : determined: the covariance matrix of the discriminating : variable space.) : : --- Performance optimisation: : : Optimal performance for Fisher discriminants is obtained for : linearly correlated Gaussian-distributed variables. Any deviation : from this ideal reduces the achievable separation power. In : particular, no discrimination at all is achieved for a variable : that has the same sample mean for signal and background, even if : the shapes of the distributions are very different. Thus, Fisher : discriminants often benefit from suitable transformations of the : input variables. For example, if a variable x in [-1,1] has a : a parabolic signal distributions, and a uniform background : distributions, their mean value is zero in both cases, leading : to no separation. The simple transformation x -> |x| renders this : variable powerful for the use in a Fisher discriminant. : : --- Performance tuning via configuration options: : : <None> : : <Suppress this message by specifying "!H" in the booking option> : ================================================================ : Fisher : Results for Fisher coefficients: : ----------------------- : Variable: Coefficient: : ----------------------- : m_jj: -0.051 : m_jjj: +0.192 : m_lv: +0.045 : m_jlv: +0.059 : m_bb: -0.211 : m_wbb: +0.549 : m_wwbb: -0.778 : (offset): +0.136 : ----------------------- : Elapsed time for training with 14000 events: 0.0139 sec Fisher : [dataset] : Evaluation of Fisher on training sample (14000 events) : Elapsed time for evaluation of 14000 events: 0.00493 sec : <CreateMVAPdfs> Separation from histogram (PDF): 0.090 (0.000) : Dataset[dataset] : Evaluation of Fisher on training sample : Creating xml weight file: dataset/weights/TMVA_Higgs_Classification_Fisher.weights.xml : Creating standalone class: dataset/weights/TMVA_Higgs_Classification_Fisher.class.C Factory : Training finished : Factory : Train method: BDT for Classification : BDT : #events: (reweighted) sig: 7000 bkg: 7000 : #events: (unweighted) sig: 7000 bkg: 7000 : Training 200 Decision Trees ... patience please : Elapsed time for training with 14000 events: 1.26 sec BDT : [dataset] : Evaluation of BDT on training sample (14000 events) : Elapsed time for evaluation of 14000 events: 0.134 sec : Creating xml weight file: dataset/weights/TMVA_Higgs_Classification_BDT.weights.xml : Creating standalone class: dataset/weights/TMVA_Higgs_Classification_BDT.class.C : Higgs_ClassificationOutput.root:/dataset/Method_BDT/BDT Factory : Training finished : Factory : Train method: DNN_CPU for Classification : : Preparing the Gaussian transformation... TFHandler_DNN_CPU : Variable Mean RMS [ Min Max ] : ----------------------------------------------------------- : m_jj: 0.0043655 0.99836 [ -3.2801 5.7307 ] : m_jjj: 0.0044371 0.99827 [ -3.2805 5.7307 ] : m_lv: 0.0054053 1.0003 [ -3.2810 5.7307 ] : m_jlv: 0.0044637 0.99837 [ -3.2803 5.7307 ] : m_bb: 0.0043676 0.99847 [ -3.2797 5.7307 ] : m_wbb: 0.0042343 0.99744 [ -3.2803 5.7307 ] : m_wwbb: 0.0046014 0.99948 [ -3.2802 5.7307 ] : ----------------------------------------------------------- : Start of deep neural network training on CPU using MT, nthreads = 1 : TFHandler_DNN_CPU : Variable Mean RMS [ Min Max ] : ----------------------------------------------------------- : m_jj: 0.0043655 0.99836 [ -3.2801 5.7307 ] : m_jjj: 0.0044371 0.99827 [ -3.2805 5.7307 ] : m_lv: 0.0054053 1.0003 [ -3.2810 5.7307 ] : m_jlv: 0.0044637 0.99837 [ -3.2803 5.7307 ] : m_bb: 0.0043676 0.99847 [ -3.2797 5.7307 ] : m_wbb: 0.0042343 0.99744 [ -3.2803 5.7307 ] : m_wwbb: 0.0046014 0.99948 [ -3.2802 5.7307 ] : ----------------------------------------------------------- : ***** Deep Learning Network ***** DEEP NEURAL NETWORK: Depth = 5 Input = ( 1, 1, 7 ) Batch size = 128 Loss function = C Layer 0 DENSE Layer: ( Input = 7 , Width = 64 ) Output = ( 1 , 128 , 64 ) Activation Function = Tanh Layer 1 DENSE Layer: ( Input = 64 , Width = 64 ) Output = ( 1 , 128 , 64 ) Activation Function = Tanh Layer 2 DENSE Layer: ( Input = 64 , Width = 64 ) Output = ( 1 , 128 , 64 ) Activation Function = Tanh Layer 3 DENSE Layer: ( Input = 64 , Width = 64 ) Output = ( 1 , 128 , 64 ) Activation Function = Tanh Layer 4 DENSE Layer: ( Input = 64 , Width = 1 ) Output = ( 1 , 128 , 1 ) Activation Function = Identity : Using 11200 events for training and 2800 for testing : Compute initial loss on the validation data : Training phase 1 of 1: Optimizer ADAM (beta1=0.9,beta2=0.999,eps=1e-07) Learning rate = 0.001 regularization 0 minimum error = 0.88466 : -------------------------------------------------------------- : Epoch | Train Err. Val. Err. t(s)/epoch t(s)/Loss nEvents/s Conv. Steps : -------------------------------------------------------------- : Start epoch iteration ... : 1 Minimum Test error found - save the configuration : 1 | 0.667137 0.621337 0.872865 0.047228 13487.8 0 : 2 Minimum Test error found - save the configuration : 2 | 0.610212 0.603286 0.716331 0.0472012 16642.5 0 : 3 Minimum Test error found - save the configuration : 3 | 0.585478 0.594042 0.909152 0.0688969 13253.1 0 : 4 Minimum Test error found - save the configuration : 4 | 0.579662 0.584781 0.756684 0.0485206 15725.2 0 : 5 | 0.571813 0.593958 0.61261 0.0472818 19698.3 1 : 6 Minimum Test error found - save the configuration : 6 | 0.569438 0.583153 0.89271 0.0833848 13759.6 0 : 7 Minimum Test error found - save the configuration : 7 | 0.563391 0.580403 0.713196 0.0536314 16883.9 0 : 8 | 0.562543 0.585883 1.42695 0.0716117 8216.4 1 : 9 | 0.55787 0.587124 0.700384 0.0541264 17231.5 2 : 10 Minimum Test error found - save the configuration : 10 | 0.55873 0.578376 0.692271 0.0496591 17329.3 0 : 11 | 0.555149 0.581201 0.871103 0.1147 14722.3 1 : 12 | 0.554282 0.584755 0.951944 0.0620945 12514.5 2 : 13 | 0.55152 0.580572 0.732036 0.0514991 16363.5 3 : 14 | 0.548805 0.586056 0.777884 0.0916716 16228.2 4 : 15 | 0.549801 0.58075 0.604655 0.0478473 19999.7 5 : 16 | 0.545369 0.58628 0.82139 0.0505959 14447.4 6 : 17 | 0.548506 0.582951 0.825033 0.0948354 15250.7 7 : 18 | 0.544446 0.581512 0.806151 0.103955 15858.8 8 : 19 | 0.543015 0.580194 0.970793 0.0555294 12167 9 : 20 | 0.543006 0.588172 0.864379 0.0487954 13654 10 : : Elapsed time for training with 14000 events: 16.6 sec : Evaluate deep neural network on CPU using batches with size = 128 : DNN_CPU : [dataset] : Evaluation of DNN_CPU on training sample (14000 events) : Elapsed time for evaluation of 14000 events: 0.41 sec : Creating xml weight file: dataset/weights/TMVA_Higgs_Classification_DNN_CPU.weights.xml : Creating standalone class: dataset/weights/TMVA_Higgs_Classification_DNN_CPU.class.C Factory : Training finished : Factory : Train method: PyKeras for Classification : : : ================================================================ : H e l p f o r M V A m e t h o d [ PyKeras ] : : : Keras is a high-level API for the Theano and Tensorflow packages. : This method wraps the training and predictions steps of the Keras : Python package for TMVA, so that dataloading, preprocessing and : evaluation can be done within the TMVA system. To use this Keras : interface, you have to generate a model with Keras first. Then, : this model can be loaded and trained in TMVA. : : : <Suppress this message by specifying "!H" in the booking option> : ================================================================ : : Split TMVA training data in 11200 training events and 2800 validation events : Training Model Summary : Option SaveBestOnly: Only model weights with smallest validation loss will be stored : Getting training history for item:0 name = 'loss' : Getting training history for item:1 name = 'accuracy' : Getting training history for item:2 name = 'val_loss' : Getting training history for item:3 name = 'val_accuracy' : Elapsed time for training with 14000 events: 33.6 sec : Setting up tf.keras : Using TensorFlow version 2 : Use Keras version from TensorFlow : tf.keras : Disabled TF eager execution when evaluating model : Loading Keras Model : Loaded model from file: trained_model_higgs.h5 PyKeras : [dataset] : Evaluation of PyKeras on training sample (14000 events) : Elapsed time for evaluation of 14000 events: 0.845 sec : Creating xml weight file: dataset/weights/TMVA_Higgs_Classification_PyKeras.weights.xml : Creating standalone class: dataset/weights/TMVA_Higgs_Classification_PyKeras.class.C Factory : Training finished : : Ranking input variables (method specific)... Likelihood : Ranking result (top variable is best ranked) : ------------------------------------- : Rank : Variable : Delta Separation : ------------------------------------- : 1 : m_bb : 3.688e-02 : 2 : m_wbb : 3.307e-02 : 3 : m_wwbb : 2.885e-02 : 4 : m_jjj : -1.155e-03 : 5 : m_jj : -1.436e-03 : 6 : m_lv : -5.963e-03 : 7 : m_jlv : -9.884e-03 : ------------------------------------- Fisher : Ranking result (top variable is best ranked) : --------------------------------- : Rank : Variable : Discr. power : --------------------------------- : 1 : m_bb : 1.279e-02 : 2 : m_wwbb : 9.131e-03 : 3 : m_wbb : 2.668e-03 : 4 : m_jlv : 9.145e-04 : 5 : m_jjj : 1.769e-04 : 6 : m_lv : 6.617e-05 : 7 : m_jj : 6.707e-06 : --------------------------------- BDT : Ranking result (top variable is best ranked) : ---------------------------------------- : Rank : Variable : Variable Importance : ---------------------------------------- : 1 : m_bb : 2.089e-01 : 2 : m_wwbb : 1.673e-01 : 3 : m_wbb : 1.568e-01 : 4 : m_jlv : 1.560e-01 : 5 : m_jjj : 1.421e-01 : 6 : m_jj : 1.052e-01 : 7 : m_lv : 6.369e-02 : ---------------------------------------- : No variable ranking supplied by classifier: DNN_CPU : No variable ranking supplied by classifier: PyKeras TH1.Print Name = TrainingHistory_DNN_CPU_trainingError, Entries= 0, Total sum= 11.3102 TH1.Print Name = TrainingHistory_DNN_CPU_valError, Entries= 0, Total sum= 11.7448 TH1.Print Name = TrainingHistory_PyKeras_'accuracy', Entries= 0, Total sum= 13.4751 TH1.Print Name = TrainingHistory_PyKeras_'loss', Entries= 0, Total sum= 11.9293 TH1.Print Name = TrainingHistory_PyKeras_'val_accuracy', Entries= 0, Total sum= 13.355 TH1.Print Name = TrainingHistory_PyKeras_'val_loss', Entries= 0, Total sum= 12.0502 Factory : === Destroy and recreate all methods via weight files for testing === : : Reading weight file: dataset/weights/TMVA_Higgs_Classification_Likelihood.weights.xml : Reading weight file: dataset/weights/TMVA_Higgs_Classification_Fisher.weights.xml : Reading weight file: dataset/weights/TMVA_Higgs_Classification_BDT.weights.xml : Reading weight file: dataset/weights/TMVA_Higgs_Classification_DNN_CPU.weights.xml : Reading weight file: dataset/weights/TMVA_Higgs_Classification_PyKeras.weights.xml
0%, time left: unknown 7%, time left: 0 sec 13%, time left: 0 sec 19%, time left: 0 sec 25%, time left: 0 sec 32%, time left: 0 sec 38%, time left: 0 sec 44%, time left: 0 sec 50%, time left: 0 sec 57%, time left: 0 sec 63%, time left: 0 sec 69%, time left: 0 sec 75%, time left: 0 sec 82%, time left: 0 sec 88%, time left: 0 sec 94%, time left: 0 sec 0%, time left: unknown 7%, time left: 0 sec 13%, time left: 0 sec 19%, time left: 0 sec 25%, time left: 0 sec 32%, time left: 0 sec 38%, time left: 0 sec 44%, time left: 0 sec 50%, time left: 0 sec 57%, time left: 0 sec 63%, time left: 0 sec 69%, time left: 0 sec 75%, time left: 0 sec 82%, time left: 0 sec 88%, time left: 0 sec 94%, time left: 0 sec 0%, time left: unknown 7%, time left: 0 sec 13%, time left: 0 sec 19%, time left: 0 sec 25%, time left: 0 sec 32%, time left: 0 sec 38%, time left: 0 sec 44%, time left: 0 sec 50%, time left: 0 sec 57%, time left: 0 sec 63%, time left: 0 sec 69%, time left: 0 sec 75%, time left: 0 sec 82%, time left: 0 sec 88%, time left: 0 sec 94%, time left: 0 sec 0%, time left: unknown 7%, time left: 0 sec 13%, time left: 0 sec 19%, time left: 0 sec 25%, time left: 0 sec 32%, time left: 0 sec 38%, time left: 0 sec 44%, time left: 0 sec 50%, time left: 0 sec 57%, time left: 0 sec 63%, time left: 0 sec 69%, time left: 0 sec 75%, time left: 0 sec 82%, time left: 0 sec 88%, time left: 0 sec 94%, time left: 0 sec 0%, time left: unknown 7%, time left: 0 sec 13%, time left: 0 sec 19%, time left: 0 sec 25%, time left: 0 sec 32%, time left: 0 sec 38%, time left: 0 sec 44%, time left: 0 sec 50%, time left: 0 sec 57%, time left: 0 sec 63%, time left: 0 sec 69%, time left: 0 sec 75%, time left: 0 sec 82%, time left: 0 sec 88%, time left: 0 sec 94%, time left: 0 sec 2024-03-19 19:21:32.392606: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:375] MLIR V1 optimization pass is not enabled 2024-03-19 19:21:32.407563: W tensorflow/c/c_api.cc:304] Operation '{name:'dense_4/bias/Assign' id:116 op device:{requested: '', assigned: ''} def:{{{node dense_4/bias/Assign}} = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](dense_4/bias, dense_4/bias/Initializer/zeros)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session. 2024-03-19 19:21:32.718620: W tensorflow/c/c_api.cc:304] Operation '{name:'dense_4/Sigmoid' id:122 op device:{requested: '', assigned: ''} def:{{{node dense_4/Sigmoid}} = Sigmoid[T=DT_FLOAT, _has_manual_control_dependencies=true](dense_4/BiasAdd)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session. 2024-03-19 19:21:32.753157: W tensorflow/c/c_api.cc:304] Operation '{name:'total/Assign' id:177 op device:{requested: '', assigned: ''} def:{{{node total/Assign}} = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](total, total/Initializer/zeros)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
Test all methods
Now we test and evaluate all methods using the test data set
factory.TestAllMethods()
factory.EvaluateAllMethods()
Factory : Test all methods Factory : Test method: Likelihood for Classification performance : Likelihood : [dataset] : Evaluation of Likelihood on testing sample (6000 events) : Elapsed time for evaluation of 6000 events: 0.0225 sec Factory : Test method: Fisher for Classification performance : Fisher : [dataset] : Evaluation of Fisher on testing sample (6000 events) : Elapsed time for evaluation of 6000 events: 0.00152 sec : Dataset[dataset] : Evaluation of Fisher on testing sample Factory : Test method: BDT for Classification performance : BDT : [dataset] : Evaluation of BDT on testing sample (6000 events) : Elapsed time for evaluation of 6000 events: 0.232 sec Factory : Test method: DNN_CPU for Classification performance : : Evaluate deep neural network on CPU using batches with size = 1000 : TFHandler_DNN_CPU : Variable Mean RMS [ Min Max ] : ----------------------------------------------------------- : m_jj: 0.017919 1.0069 [ -3.3498 3.4247 ] : m_jjj: 0.020352 1.0044 [ -3.2831 3.3699 ] : m_lv: 0.016356 0.99266 [ -3.2339 3.3958 ] : m_jlv: -0.018431 0.98242 [ -3.0632 5.7307 ] : m_bb: 0.0069564 0.98851 [ -2.9734 3.3513 ] : m_wbb: -0.010633 0.99340 [ -3.2442 3.2244 ] : m_wwbb: -0.012669 0.99259 [ -3.1871 5.7307 ] : ----------------------------------------------------------- DNN_CPU : [dataset] : Evaluation of DNN_CPU on testing sample (6000 events) : Elapsed time for evaluation of 6000 events: 0.135 sec Factory : Test method: PyKeras for Classification performance : : Setting up tf.keras : Using TensorFlow version 2 : Use Keras version from TensorFlow : tf.keras : Disabled TF eager execution when evaluating model : Loading Keras Model : Loaded model from file: trained_model_higgs.h5 PyKeras : [dataset] : Evaluation of PyKeras on testing sample (6000 events) : Elapsed time for evaluation of 6000 events: 0.923 sec Factory : Evaluate all methods Factory : Evaluate classifier: Likelihood : Likelihood : [dataset] : Loop over test events and fill histograms with classifier response... : TFHandler_Likelihood : Variable Mean RMS [ Min Max ] : ----------------------------------------------------------- : m_jj: 1.0447 0.66216 [ 0.14661 10.222 ] : m_jjj: 1.0275 0.37015 [ 0.34201 5.6016 ] : m_lv: 1.0500 0.15582 [ 0.29757 2.8989 ] : m_jlv: 1.0053 0.39478 [ 0.41660 5.8799 ] : m_bb: 0.97464 0.52138 [ 0.10941 5.5163 ] : m_wbb: 1.0296 0.35719 [ 0.38878 3.9747 ] : m_wwbb: 0.95617 0.30368 [ 0.44118 4.0728 ] : ----------------------------------------------------------- Factory : Evaluate classifier: Fisher : Fisher : [dataset] : Loop over test events and fill histograms with classifier response... : : Also filling probability and rarity histograms (on request)... TFHandler_Fisher : Variable Mean RMS [ Min Max ] : ----------------------------------------------------------- : m_jj: 1.0447 0.66216 [ 0.14661 10.222 ] : m_jjj: 1.0275 0.37015 [ 0.34201 5.6016 ] : m_lv: 1.0500 0.15582 [ 0.29757 2.8989 ] : m_jlv: 1.0053 0.39478 [ 0.41660 5.8799 ] : m_bb: 0.97464 0.52138 [ 0.10941 5.5163 ] : m_wbb: 1.0296 0.35719 [ 0.38878 3.9747 ] : m_wwbb: 0.95617 0.30368 [ 0.44118 4.0728 ] : ----------------------------------------------------------- Factory : Evaluate classifier: BDT : BDT : [dataset] : Loop over test events and fill histograms with classifier response... : TFHandler_BDT : Variable Mean RMS [ Min Max ] : ----------------------------------------------------------- : m_jj: 1.0447 0.66216 [ 0.14661 10.222 ] : m_jjj: 1.0275 0.37015 [ 0.34201 5.6016 ] : m_lv: 1.0500 0.15582 [ 0.29757 2.8989 ] : m_jlv: 1.0053 0.39478 [ 0.41660 5.8799 ] : m_bb: 0.97464 0.52138 [ 0.10941 5.5163 ] : m_wbb: 1.0296 0.35719 [ 0.38878 3.9747 ] : m_wwbb: 0.95617 0.30368 [ 0.44118 4.0728 ] : ----------------------------------------------------------- Factory : Evaluate classifier: DNN_CPU : DNN_CPU : [dataset] : Loop over test events and fill histograms with classifier response... : : Evaluate deep neural network on CPU using batches with size = 1000 : TFHandler_DNN_CPU : Variable Mean RMS [ Min Max ] : ----------------------------------------------------------- : m_jj: 0.0043655 0.99836 [ -3.2801 5.7307 ] : m_jjj: 0.0044371 0.99827 [ -3.2805 5.7307 ] : m_lv: 0.0054053 1.0003 [ -3.2810 5.7307 ] : m_jlv: 0.0044637 0.99837 [ -3.2803 5.7307 ] : m_bb: 0.0043676 0.99847 [ -3.2797 5.7307 ] : m_wbb: 0.0042343 0.99744 [ -3.2803 5.7307 ] : m_wwbb: 0.0046014 0.99948 [ -3.2802 5.7307 ] : ----------------------------------------------------------- TFHandler_DNN_CPU : Variable Mean RMS [ Min Max ] : ----------------------------------------------------------- : m_jj: 0.017919 1.0069 [ -3.3498 3.4247 ] : m_jjj: 0.020352 1.0044 [ -3.2831 3.3699 ] : m_lv: 0.016356 0.99266 [ -3.2339 3.3958 ] : m_jlv: -0.018431 0.98242 [ -3.0632 5.7307 ] : m_bb: 0.0069564 0.98851 [ -2.9734 3.3513 ] : m_wbb: -0.010633 0.99340 [ -3.2442 3.2244 ] : m_wwbb: -0.012669 0.99259 [ -3.1871 5.7307 ] : ----------------------------------------------------------- Factory : Evaluate classifier: PyKeras : PyKeras : [dataset] : Loop over test events and fill histograms with classifier response... : TFHandler_PyKeras : Variable Mean RMS [ Min Max ] : ----------------------------------------------------------- : m_jj: 1.0447 0.66216 [ 0.14661 10.222 ] : m_jjj: 1.0275 0.37015 [ 0.34201 5.6016 ] : m_lv: 1.0500 0.15582 [ 0.29757 2.8989 ] : m_jlv: 1.0053 0.39478 [ 0.41660 5.8799 ] : m_bb: 0.97464 0.52138 [ 0.10941 5.5163 ] : m_wbb: 1.0296 0.35719 [ 0.38878 3.9747 ] : m_wwbb: 0.95617 0.30368 [ 0.44118 4.0728 ] : ----------------------------------------------------------- : : Evaluation results ranked by best signal efficiency and purity (area) : ------------------------------------------------------------------------------------------------------------------- : DataSet MVA : Name: Method: ROC-integ : dataset DNN_CPU : 0.758 : dataset PyKeras : 0.757 : dataset BDT : 0.754 : dataset Likelihood : 0.698 : dataset Fisher : 0.642 : ------------------------------------------------------------------------------------------------------------------- : : Testing efficiency compared to training efficiency (overtraining check) : ------------------------------------------------------------------------------------------------------------------- : DataSet MVA Signal efficiency: from test sample (from training sample) : Name: Method: @B=0.01 @B=0.10 @B=0.30 : ------------------------------------------------------------------------------------------------------------------- : dataset DNN_CPU : 0.135 (0.136) 0.407 (0.427) 0.672 (0.700) : dataset PyKeras : 0.125 (0.105) 0.403 (0.418) 0.672 (0.679) : dataset BDT : 0.098 (0.099) 0.393 (0.402) 0.657 (0.681) : dataset Likelihood : 0.085 (0.082) 0.355 (0.363) 0.580 (0.596) : dataset Fisher : 0.015 (0.015) 0.121 (0.131) 0.487 (0.506) : ------------------------------------------------------------------------------------------------------------------- : Dataset:dataset : Created tree 'TestTree' with 6000 events : Dataset:dataset : Created tree 'TrainTree' with 14000 events : Factory : Thank you for using TMVA! : For citation information, please visit: http://tmva.sf.net/citeTMVA.html
0%, time left: unknown 7%, time left: 0 sec 13%, time left: 0 sec 19%, time left: 0 sec 25%, time left: 0 sec 32%, time left: 0 sec 38%, time left: 0 sec 44%, time left: 0 sec 50%, time left: 0 sec 57%, time left: 0 sec 63%, time left: 0 sec 69%, time left: 0 sec 75%, time left: 0 sec 82%, time left: 0 sec 88%, time left: 0 sec 94%, time left: 0 sec 0%, time left: unknown 7%, time left: 0 sec 13%, time left: 0 sec 19%, time left: 0 sec 25%, time left: 0 sec 32%, time left: 0 sec 38%, time left: 0 sec 44%, time left: 0 sec 50%, time left: 0 sec 57%, time left: 0 sec 63%, time left: 0 sec 69%, time left: 0 sec 75%, time left: 0 sec 82%, time left: 0 sec 88%, time left: 0 sec 94%, time left: 0 sec 0%, time left: unknown 7%, time left: 0 sec 13%, time left: 0 sec 19%, time left: 0 sec 25%, time left: 0 sec 32%, time left: 0 sec 38%, time left: 0 sec 44%, time left: 0 sec 50%, time left: 0 sec 57%, time left: 0 sec 63%, time left: 0 sec 69%, time left: 0 sec 75%, time left: 0 sec 82%, time left: 0 sec 88%, time left: 0 sec 94%, time left: 0 sec 0%, time left: unknown 7%, time left: 0 sec 13%, time left: 0 sec 19%, time left: 0 sec 25%, time left: 0 sec 32%, time left: 0 sec 38%, time left: 0 sec 44%, time left: 0 sec 50%, time left: 0 sec 57%, time left: 0 sec 63%, time left: 0 sec 69%, time left: 0 sec 75%, time left: 0 sec 82%, time left: 0 sec 88%, time left: 0 sec 94%, time left: 0 sec 2024-03-19 19:21:34.626162: W tensorflow/c/c_api.cc:304] Operation '{name:'dense_2_1/bias/Assign' id:476 op device:{requested: '', assigned: ''} def:{{{node dense_2_1/bias/Assign}} = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](dense_2_1/bias, dense_2_1/bias/Initializer/zeros)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session. 2024-03-19 19:21:34.894727: W tensorflow/c/c_api.cc:304] Operation '{name:'dense_4_1/Sigmoid' id:530 op device:{requested: '', assigned: ''} def:{{{node dense_4_1/Sigmoid}} = Sigmoid[T=DT_FLOAT, _has_manual_control_dependencies=true](dense_4_1/BiasAdd)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session. 2024-03-19 19:21:34.923031: W tensorflow/c/c_api.cc:304] Operation '{name:'total_1/Assign' id:585 op device:{requested: '', assigned: ''} def:{{{node total_1/Assign}} = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](total_1, total_1/Initializer/zeros)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
after we get the ROC curve and we display
c1 = factory.GetROCCurve(loader)
c1.Draw()
at the end we close the output file which contains the evaluation result of all methods and it can be used by TMVAGUI to display additional plots
outputFile.Close()
Draw all canvases
from ROOT import gROOT
gROOT.GetListOfCanvases().Draw()