created by Doug Williams (https://github.com/williamsdoug) and Ignacio Oguiza (https://github.com/timeseriesAI/tsai).
contact: timeseriesAI@gmail.com
The purpose of this notebook is to demonstrate both multi-class and multi-label classification using tsai
.
Multi-class classification: While the output can take on multiple possible values, for any given sample the output can take on only a single value. In other words, the label values are mutually exclusive. Implication:
CategoricalCrossEntropy (nn.CrossEntropyLoss
in Pytorch, CrossEntropyLossFlat
in fastai) is used as the loss function during training
Softmax is used to determine prediction since only one label value can be true, the predicted label is the value label value with the greatest probability.
Softmax reduces the potential for spurious label predictions since only the label with the highest probability is selected
In both Pytorch and fastai the loss combines a Softmax layer and the CrossEntropyLoss in one single class, so Softmax shouldn't be added to the model.
Multi-label classification: This is the more general case where an individual sample can have one or more labels, relaxing the mutual label exclusivity constraint. Implications:
BinaryCrossEntropy (nn.BCEWithLogitsLoss
in Pytorch, BCEWithLogitsLossFlat
in fastai) is used as the loss function during training
Sigmoid is used to determine prediction since multiple labels may be true. In both Pytorch and fastai the loss combines a Sigmoid layer and the BCELoss in one single class, so Sigmoid shouldn't be added to the model.
Relative to multi-class classification, multi-label classification may be more prone to spurious false-positive labels.
# ## NOTE: UNCOMMENT AND RUN THIS CELL IF YOU NEED TO INSTALL/ UPGRADE TSAI
# stable = False # True: latest version from github, False: stable version in pip
# if stable:
# !pip install -Uqq tsai
# else:
# !pip install -Uqq git+https://github.com/timeseriesAI/tsai.git
# ## NOTE: REMEMBER TO RESTART YOUR RUNTIME ONCE THE INSTALLATION IS FINISHED
from tsai.all import *
print('tsai :', tsai.__version__)
print('fastai :', fastai.__version__)
print('fastcore :', fastcore.__version__)
print('torch :', torch.__version__)
tsai : 0.2.18 fastai : 2.3.1 fastcore : 1.3.19 torch : 1.8.1+cu102
For this example we will be using the UCR ECG5000 heartbeat dataset with is based on the Physionet BIDMC Congestive Heart Failure Database, specifically recording chf07.
For the purposes of this example the UCR labels will be mapped to more descriptive labels:
class_map = {
'1':'Nor', # N:1 - Normal
'2':'RoT', # r:2 - R-on-T premature ventricular contraction
'3':'PVC', # V:3 - Premature ventricular contraction
'4':'SPC', # S:4 - Supraventricular premature or ectopic beat (atrial or nodal)
'5':'Unk', # Q:5 - Unclassifiable beat
}
class_map
{'1': 'Nor', '2': 'RoT', '3': 'PVC', '4': 'SPC', '5': 'Unk'}
# dataset id
dsid = 'ECG5000'
X, y, splits = get_UCR_data(dsid, split_data=False)
labeler = ReLabeler(class_map)
new_y = labeler(y) # map to more descriptive labels
X.shape, new_y.shape, splits, new_y
((5000, 1, 140), (5000,), ((#500) [0,1,2,3,4,5,6,7,8,9...], (#4500) [500,501,502,503,504,505,506,507,508,509...]), array(['Nor', 'Nor', 'Nor', ..., 'RoT', 'RoT', 'RoT'], dtype='<U3'))
label_counts = collections.Counter(new_y)
print('Counts by label:', dict(label_counts))
print(f'Naive Accuracy: {100*max(label_counts.values())/sum(label_counts.values()):0.2f}%')
Counts by label: {'Nor': 2919, 'RoT': 1767, 'PVC': 96, 'SPC': 194, 'Unk': 24} Naive Accuracy: 58.38%
Note: naive accuracy is calculated by assuming all samples are predicted as 'Nor' (most frequent label).
tfms = [None, TSClassification()] # TSClassification == Categorize
batch_tfms = TSStandardize()
dls = get_ts_dls(X, new_y, splits=splits, tfms=tfms, batch_tfms=batch_tfms, bs=[64, 128])
dls.dataset
(#500) [(TSTensor(vars:1, len:140), TensorCategory(0)),(TSTensor(vars:1, len:140), TensorCategory(0)),(TSTensor(vars:1, len:140), TensorCategory(0)),(TSTensor(vars:1, len:140), TensorCategory(0)),(TSTensor(vars:1, len:140), TensorCategory(0)),(TSTensor(vars:1, len:140), TensorCategory(0)),(TSTensor(vars:1, len:140), TensorCategory(0)),(TSTensor(vars:1, len:140), TensorCategory(0)),(TSTensor(vars:1, len:140), TensorCategory(0)),(TSTensor(vars:1, len:140), TensorCategory(0))...]
dls.show_batch(sharey=True)
There are at least 2 equivalent ways to build a time series learner:
model = build_ts_model(InceptionTimePlus, dls=dls)
learn = Learner(dls, model, metrics=accuracy)
learn = ts_learner(dls, metrics=accuracy) # == ts_learner(dls, arch=InceptionTimePlus, metrics=accuracy) since InceptionTimePlus is the default arch
learn.lr_find(suggestions=False)
learn = ts_learner(dls, metrics=accuracy, cbs=ShowGraph())
learn.fit_one_cycle(10, lr_max=1e-3)
epoch | train_loss | valid_loss | accuracy | time |
---|---|---|---|---|
0 | 1.262671 | 1.530813 | 0.583778 | 00:00 |
1 | 0.890005 | 1.376411 | 0.874444 | 00:00 |
2 | 0.664608 | 0.953245 | 0.877556 | 00:00 |
3 | 0.538610 | 0.400819 | 0.923778 | 00:00 |
4 | 0.453108 | 0.271983 | 0.933111 | 00:00 |
5 | 0.392681 | 0.245539 | 0.936444 | 00:00 |
6 | 0.345966 | 0.309330 | 0.925778 | 00:00 |
7 | 0.308728 | 0.216228 | 0.937778 | 00:00 |
8 | 0.279623 | 0.218552 | 0.940000 | 00:00 |
9 | 0.259124 | 0.215775 | 0.943111 | 00:00 |
learn.show_results(sharey=True)
learn.show_probas()
interp = ClassificationInterpretation.from_learner(learn)
interp.plot_confusion_matrix()
interp.most_confused(min_val=3)
[('SPC', 'RoT', 108), ('PVC', 'RoT', 42), ('PVC', 'Nor', 27), ('SPC', 'Nor', 18), ('RoT', 'SPC', 17), ('RoT', 'Nor', 13), ('Unk', 'Nor', 12), ('Unk', 'RoT', 10), ('Nor', 'RoT', 7)]
learn.dls.vocab
['Nor', 'PVC', 'RoT', 'SPC', 'Unk']
# if using ClassificationInterpretation
interp.vocab
['Nor', 'PVC', 'RoT', 'SPC', 'Unk']
X_test , y_test = X[splits[1]], new_y[splits[1]]
valid_dl = dls.valid
test_ds = valid_dl.dataset.add_test(X_test, y_test)
test_dl = valid_dl.new(test_ds)
vocab = learn.dls.vocab
_, temp_targets, temp_preds = learn.get_preds(dl=test_dl, with_decoded=True, save_preds=None, save_targs=None)
temp_preds
tensor([0, 0, 0, ..., 2, 2, 3])
Predictions are returned as an array of integers, one per row, containing the index of the associated value
decoded_preds = vocab[temp_preds]
decoded_preds
(#4500) ['Nor','Nor','Nor','Nor','Nor','Nor','Nor','Nor','Nor','Nor'...]
decoded_preds[-10:]
(#10) ['RoT','RoT','RoT','SPC','SPC','RoT','SPC','RoT','RoT','SPC']
# alternative decoder that works for both multi-label and multi-class
decoded_preds = L(vocab[p] for p in temp_preds)
decoded_preds
(#4500) ['Nor','Nor','Nor','Nor','Nor','Nor','Nor','Nor','Nor','Nor'...]
decoded_preds[-10:]
(#10) ['RoT','RoT','RoT','SPC','SPC','RoT','SPC','RoT','RoT','SPC']
Note: While in this example the new Pre label is a composite of existing labels, more typically multi-label classification problems include orthogonal label groups. For example in ECG classification one might have labels related to timing (e.g.: premature), QRS shape (e.g.: block) and other factors (e.g.: ST elevation or depression)
class_map = {
'1':['Nor'], # N:1 - Normal
'2':['RoT', 'Pre'], # r:2 - R-on-T premature ventricular contraction
'3':['PVC', 'Pre'] , # V:3 - Premature ventricular contraction
'4':['SPC', 'Pre'], # S:4 - Supraventricular premature or ectopic beat (atrial or nodal)
'5':['Unk'], # Q:5 - Unclassifiable beat
}
class_map
{'1': ['Nor'], '2': ['RoT', 'Pre'], '3': ['PVC', 'Pre'], '4': ['SPC', 'Pre'], '5': ['Unk']}
labeler = ReLabeler(class_map)
y_multi = labeler(y)
y_multi
array([list(['Nor']), list(['Nor']), list(['Nor']), ..., list(['RoT', 'Pre']), list(['RoT', 'Pre']), list(['RoT', 'Pre'])], dtype=object)
label_counts = collections.Counter([a for r in y_multi for a in r])
print('Counts by label:', dict(label_counts))
Counts by label: {'Nor': 2919, 'RoT': 1767, 'Pre': 2057, 'PVC': 96, 'SPC': 194, 'Unk': 24}
tfms = [None, [TSClassification()]]
with tfms = [None, TSMultiLabelClassification()]
tfms = [None, TSMultiLabelClassification()] # TSMultiLabelClassification() == [MultiCategorize(), OneHotEncode()]
batch_tfms = TSStandardize()
dls = get_ts_dls(X, y_multi, splits=splits, tfms=tfms, batch_tfms=batch_tfms, bs=[64, 128])
dls.dataset
(#500) [(TSTensor(vars:1, len:140), TensorMultiCategory([1., 0., 0., 0., 0., 0.])),(TSTensor(vars:1, len:140), TensorMultiCategory([1., 0., 0., 0., 0., 0.])),(TSTensor(vars:1, len:140), TensorMultiCategory([1., 0., 0., 0., 0., 0.])),(TSTensor(vars:1, len:140), TensorMultiCategory([1., 0., 0., 0., 0., 0.])),(TSTensor(vars:1, len:140), TensorMultiCategory([1., 0., 0., 0., 0., 0.])),(TSTensor(vars:1, len:140), TensorMultiCategory([1., 0., 0., 0., 0., 0.])),(TSTensor(vars:1, len:140), TensorMultiCategory([1., 0., 0., 0., 0., 0.])),(TSTensor(vars:1, len:140), TensorMultiCategory([1., 0., 0., 0., 0., 0.])),(TSTensor(vars:1, len:140), TensorMultiCategory([1., 0., 0., 0., 0., 0.])),(TSTensor(vars:1, len:140), TensorMultiCategory([1., 0., 0., 0., 0., 0.]))...]
dls.show_batch(sharey=True)
Use metrics=accuracy_multi
in place of metrics=accuracy
used in earlier multi-class example
accuracy_multi can be calculated by sample (default) or by predicted label. When using accuracy_multi by_sample=True, all predicted labels for each sample must be correct to be counted as a correct sample. This is sometimes difficult (when having too many labels). There are cases when labels are partially correct. If we want to account for these, we'll set by_sample to False.
def accuracy_multi(inp, targ, thresh=0.5, sigmoid=True, by_sample=False):
"Computes accuracy when `inp` and `targ` are the same size."
if sigmoid: inp = inp.sigmoid()
correct = (inp>thresh)==targ.bool()
if by_sample:
return (correct.float().mean(-1) == 1).float().mean()
else:
inp,targ = flatten_check(inp,targ)
return correct.float().mean()
learn = ts_learner(dls, InceptionTime, metrics=accuracy_multi)
learn.lr_find(suggestions=False)
learn = ts_learner(dls, InceptionTimePlus, metrics=[partial(accuracy_multi, by_sample=True), partial(accuracy_multi, by_sample=False)], cbs=ShowGraph())
learn.fit_one_cycle(10, lr_max=1e-3)
epoch | train_loss | valid_loss | accuracy_multi | accuracy_multi | time |
---|---|---|---|---|---|
0 | 0.684776 | 0.695255 | 0.000000 | 0.557148 | 00:00 |
1 | 0.572439 | 0.644878 | 0.444222 | 0.814593 | 00:00 |
2 | 0.471255 | 0.441004 | 0.925111 | 0.972556 | 00:00 |
3 | 0.392487 | 0.231447 | 0.920889 | 0.969630 | 00:00 |
4 | 0.331273 | 0.158744 | 0.926667 | 0.973296 | 00:00 |
5 | 0.284419 | 0.144118 | 0.917556 | 0.971185 | 00:00 |
6 | 0.248883 | 0.122650 | 0.926444 | 0.974148 | 00:00 |
7 | 0.220407 | 0.111970 | 0.928000 | 0.974704 | 00:00 |
8 | 0.198430 | 0.113578 | 0.927778 | 0.975148 | 00:00 |
9 | 0.180306 | 0.110633 | 0.927556 | 0.975148 | 00:00 |
A naive classifier that always predicts no true labels would achieve 76% accuracy, so the classifier should do much better. This also demonstrates the weakness of relying overly on the accuracy metric (with by_sample=False) since due to the prevalance of false outputs.
label_counts = collections.Counter([a for r in y_multi for a in r])
print(f'Naive Accuracy: {100*(1-sum(label_counts.values())/(len(y_multi)*len(label_counts))):0.2f}%')
Naive Accuracy: 76.48%
def precision_multi(inp, targ, thresh=0.5, sigmoid=True):
"Computes precision when `inp` and `targ` are the same size."
inp,targ = flatten_check(inp,targ)
if sigmoid: inp = inp.sigmoid()
pred = inp>thresh
correct = pred==targ.bool()
TP = torch.logical_and(correct, (targ==1).bool()).sum()
FP = torch.logical_and(~correct, (targ==0).bool()).sum()
precision = TP/(TP+FP)
return precision
def recall_multi(inp, targ, thresh=0.5, sigmoid=True):
"Computes recall when `inp` and `targ` are the same size."
inp,targ = flatten_check(inp,targ)
if sigmoid: inp = inp.sigmoid()
pred = inp>thresh
correct = pred==targ.bool()
TP = torch.logical_and(correct, (targ==1).bool()).sum()
FN = torch.logical_and(~correct, (targ==1).bool()).sum()
recall = TP/(TP+FN)
return recall
def specificity_multi(inp, targ, thresh=0.5, sigmoid=True):
"Computes specificity (true negative rate) when `inp` and `targ` are the same size."
inp,targ = flatten_check(inp,targ)
if sigmoid: inp = inp.sigmoid()
pred = inp>thresh
correct = pred==targ.bool()
TN = torch.logical_and(correct, (targ==0).bool()).sum()
FP = torch.logical_and(~correct, (targ==0).bool()).sum()
specificity = TN/(TN+FP)
return specificity
def balanced_accuracy_multi(inp, targ, thresh=0.5, sigmoid=True):
"Computes balanced accuracy when `inp` and `targ` are the same size."
inp,targ = flatten_check(inp,targ)
if sigmoid: inp = inp.sigmoid()
pred = inp>thresh
correct = pred==targ.bool()
TP = torch.logical_and(correct, (targ==1).bool()).sum()
TN = torch.logical_and(correct, (targ==0).bool()).sum()
FN = torch.logical_and(~correct, (targ==1).bool()).sum()
FP = torch.logical_and(~correct, (targ==0).bool()).sum()
TPR = TP/(TP+FN)
TNR = TN/(TN+FP)
balanced_accuracy = (TPR+TNR)/2
return balanced_accuracy
def Fbeta_multi(inp, targ, beta=1.0, thresh=0.5, sigmoid=True):
"Computes Fbeta when `inp` and `targ` are the same size."
inp,targ = flatten_check(inp,targ)
if sigmoid: inp = inp.sigmoid()
pred = inp>thresh
correct = pred==targ.bool()
TP = torch.logical_and(correct, (targ==1).bool()).sum()
TN = torch.logical_and(correct, (targ==0).bool()).sum()
FN = torch.logical_and(~correct, (targ==1).bool()).sum()
FP = torch.logical_and(~correct, (targ==0).bool()).sum()
precision = TP/(TP+FP)
recall = TP/(TP+FN)
beta2 = beta*beta
if precision+recall > 0:
Fbeta = (1+beta2)*precision*recall/(beta2*precision+recall)
else:
Fbeta = 0
return Fbeta
def F1_multi(*args, **kwargs):
return Fbeta_multi(*args, **kwargs) # beta defaults to 1.0
metrics =[accuracy_multi, balanced_accuracy_multi, precision_multi, recall_multi, specificity_multi, F1_multi]
learn = ts_learner(dls, InceptionTime, metrics=metrics, cbs=ShowGraph())
learn.fit_one_cycle(10, lr_max=1e-3)
epoch | train_loss | valid_loss | accuracy_multi | balanced_accuracy_multi | precision_multi | recall_multi | specificity_multi | F1_multi | time |
---|---|---|---|---|---|---|---|---|---|
0 | 0.619552 | 0.676024 | 0.616222 | 0.468761 | 0.232413 | 0.218974 | 0.718547 | 0.225265 | 00:00 |
1 | 0.523001 | 0.643696 | 0.922889 | 0.884351 | 0.938625 | 0.787276 | 0.981427 | 0.848500 | 00:00 |
2 | 0.427766 | 0.461396 | 0.971222 | 0.963247 | 0.952847 | 0.943780 | 0.982714 | 0.947991 | 00:00 |
3 | 0.352478 | 0.234231 | 0.973778 | 0.968561 | 0.959408 | 0.954828 | 0.982295 | 0.957061 | 00:00 |
4 | 0.296719 | 0.150781 | 0.974704 | 0.968764 | 0.966436 | 0.952417 | 0.985112 | 0.959157 | 00:00 |
5 | 0.253697 | 0.118545 | 0.974667 | 0.969775 | 0.963678 | 0.956106 | 0.983443 | 0.959811 | 00:00 |
6 | 0.220356 | 0.118755 | 0.975148 | 0.969557 | 0.965882 | 0.954321 | 0.984794 | 0.959948 | 00:00 |
7 | 0.196382 | 0.113336 | 0.975370 | 0.970159 | 0.965123 | 0.955897 | 0.984420 | 0.960397 | 00:00 |
8 | 0.176770 | 0.098999 | 0.975556 | 0.969992 | 0.967252 | 0.954542 | 0.985443 | 0.960672 | 00:00 |
9 | 0.160750 | 0.098818 | 0.975556 | 0.969936 | 0.967457 | 0.954319 | 0.985553 | 0.960640 | 00:00 |
tsai
automatically calculates class weights. It's a dataloaders attribute called cws. You should use dls.train.cws to avoid any leakage.tfms = [None, TSMultiLabelClassification()] # TSMultiLabelClassification() == [MultiCategorize(), OneHotEncode()]
batch_tfms = TSStandardize()
dls = get_ts_dls(X, y_multi, splits=splits, tfms=tfms, batch_tfms=batch_tfms, bs=[64, 128])
dls.vocab, dls.train.cws
(['Nor', 'PVC', 'Pre', 'RoT', 'SPC', 'Unk'], tensor([ 0.7123, 49.0000, 1.4272, 1.8249, 25.3158, 249.0000], device='cuda:0'))
loss_func
metrics = [accuracy_multi, balanced_accuracy_multi, precision_multi, recall_multi, specificity_multi, F1_multi]
learn = ts_learner(dls, InceptionTime, metrics=metrics, loss_func=BCEWithLogitsLossFlat(pos_weight=dls.train.cws), cbs=ShowGraph())
learn.lr_find(suggestions=True, start_lr=1e-06, end_lr=1e2)
SuggestedLRs(lr_min=0.007585775852203369, lr_steep=0.0019054607255384326)
learn.fit_one_cycle(20, lr_max=1e-3)
epoch | train_loss | valid_loss | accuracy_multi | balanced_accuracy_multi | precision_multi | recall_multi | specificity_multi | F1_multi | time |
---|---|---|---|---|---|---|---|---|---|
0 | 1.091189 | 1.084347 | 0.422630 | 0.352525 | 0.165868 | 0.251620 | 0.453430 | 0.199765 | 00:00 |
1 | 1.020666 | 1.082213 | 0.417111 | 0.353084 | 0.165968 | 0.268336 | 0.437832 | 0.204948 | 00:00 |
2 | 0.922636 | 1.058951 | 0.402370 | 0.388624 | 0.206054 | 0.418439 | 0.358809 | 0.275992 | 00:00 |
3 | 0.835018 | 0.927770 | 0.819333 | 0.837387 | 0.651673 | 0.912464 | 0.762310 | 0.746524 | 00:00 |
4 | 0.745589 | 0.747329 | 0.875111 | 0.897848 | 0.747842 | 0.966386 | 0.829310 | 0.833336 | 00:00 |
5 | 0.669781 | 0.909769 | 0.691259 | 0.757690 | 0.411902 | 0.901758 | 0.613622 | 0.562675 | 00:00 |
6 | 0.606268 | 0.768700 | 0.915408 | 0.924675 | 0.813711 | 0.949538 | 0.899812 | 0.874018 | 00:00 |
7 | 0.560813 | 0.764535 | 0.964518 | 0.965174 | 0.927688 | 0.967743 | 0.962605 | 0.946712 | 00:00 |
8 | 0.512843 | 0.617982 | 0.859704 | 0.889962 | 0.690358 | 0.967510 | 0.812414 | 0.801244 | 00:00 |
9 | 0.469022 | 0.594738 | 0.919370 | 0.931957 | 0.819229 | 0.966368 | 0.897545 | 0.883189 | 00:00 |
10 | 0.431915 | 0.720040 | 0.960852 | 0.962464 | 0.918961 | 0.967672 | 0.957256 | 0.941817 | 00:00 |
11 | 0.397774 | 0.717876 | 0.949074 | 0.956428 | 0.888446 | 0.978097 | 0.934759 | 0.928229 | 00:00 |
12 | 0.368667 | 0.725767 | 0.947630 | 0.954502 | 0.883777 | 0.975511 | 0.933493 | 0.924511 | 00:00 |
13 | 0.341911 | 0.701678 | 0.929630 | 0.942476 | 0.850208 | 0.981492 | 0.903460 | 0.905056 | 00:00 |
14 | 0.316593 | 0.699334 | 0.956000 | 0.960952 | 0.900017 | 0.976422 | 0.945481 | 0.934620 | 00:00 |
15 | 0.294102 | 0.697452 | 0.955037 | 0.960590 | 0.902035 | 0.976873 | 0.944307 | 0.935834 | 00:00 |
16 | 0.273928 | 0.698898 | 0.955556 | 0.960723 | 0.904848 | 0.976088 | 0.945358 | 0.936990 | 00:00 |
17 | 0.256562 | 0.700821 | 0.956407 | 0.961294 | 0.906375 | 0.975940 | 0.946649 | 0.937835 | 00:00 |
18 | 0.242875 | 0.705629 | 0.955556 | 0.961174 | 0.903288 | 0.977846 | 0.944503 | 0.936805 | 00:00 |
19 | 0.229243 | 0.706437 | 0.953037 | 0.959426 | 0.898047 | 0.978293 | 0.940560 | 0.933833 | 00:00 |
metrics = [accuracy_multi, balanced_accuracy_multi, precision_multi, recall_multi, specificity_multi, F1_multi]
learn = ts_learner(dls, InceptionTime, metrics=metrics, loss_func=BCEWithLogitsLossFlat(pos_weight=dls.train.cws.sqrt()), cbs=ShowGraph())
learn.lr_find(suggestions=True, start_lr=1e-06, end_lr=1e2)
SuggestedLRs(lr_min=0.12022644281387329, lr_steep=0.006918309722095728)
learn.fit_one_cycle(10, lr_max=1e-3)
epoch | train_loss | valid_loss | accuracy_multi | balanced_accuracy_multi | precision_multi | recall_multi | specificity_multi | F1_multi | time |
---|---|---|---|---|---|---|---|---|---|
0 | 0.688609 | 0.728579 | 0.532630 | 0.473540 | 0.267852 | 0.408822 | 0.538259 | 0.323407 | 00:00 |
1 | 0.597962 | 0.696789 | 0.678852 | 0.559356 | 0.280636 | 0.397362 | 0.721351 | 0.328867 | 00:00 |
2 | 0.515562 | 0.563852 | 0.947963 | 0.929882 | 0.865340 | 0.898346 | 0.961417 | 0.881244 | 00:00 |
3 | 0.441040 | 0.384785 | 0.973037 | 0.967770 | 0.951182 | 0.955228 | 0.980312 | 0.953138 | 00:00 |
4 | 0.384890 | 0.301475 | 0.971852 | 0.965373 | 0.946298 | 0.951179 | 0.979567 | 0.948707 | 00:00 |
5 | 0.341611 | 0.224056 | 0.975926 | 0.971010 | 0.965443 | 0.957608 | 0.984413 | 0.961380 | 00:00 |
6 | 0.304272 | 0.223900 | 0.977185 | 0.973695 | 0.962978 | 0.964983 | 0.982407 | 0.963853 | 00:00 |
7 | 0.271627 | 0.206188 | 0.976741 | 0.974147 | 0.960167 | 0.967426 | 0.980868 | 0.963644 | 00:00 |
8 | 0.247647 | 0.195757 | 0.975000 | 0.971378 | 0.960103 | 0.961454 | 0.981302 | 0.960739 | 00:00 |
9 | 0.227496 | 0.193261 | 0.975445 | 0.971940 | 0.960373 | 0.962463 | 0.981417 | 0.961375 | 00:00 |
X_test , y_test = X[splits[1]], y_multi[splits[1]]
valid_dl = dls.valid
test_ds = valid_dl.dataset.add_test(X_test, y_test)
test_dl = valid_dl.new(test_ds)
vocab = learn.dls.vocab
vocab
['Nor', 'PVC', 'Pre', 'RoT', 'SPC', 'Unk']
learn.dls.vocab
['Nor', 'PVC', 'Pre', 'RoT', 'SPC', 'Unk']
_, temp_targets, temp_preds = learn.get_preds(dl=test_dl, with_decoded=True, save_preds=None, save_targs=None)
temp_targets
TensorMultiCategory([[1., 0., 0., 0., 0., 0.], [1., 0., 0., 0., 0., 0.], [1., 0., 0., 0., 0., 0.], ..., [0., 0., 1., 1., 0., 0.], [0., 0., 1., 1., 0., 0.], [0., 0., 1., 1., 0., 0.]])
temp_preds
tensor([[ True, False, False, False, False, False], [ True, False, False, False, False, False], [ True, False, False, False, False, False], ..., [False, False, True, True, False, False], [False, False, True, True, False, False], [False, False, True, False, True, False]])
Predictions are returned as a per-row binary array where True values correspond to predited classes
decoded_preds = L(vocab[p] for p in temp_preds)
decoded_preds
(#4500) [['Nor'],['Nor'],['Nor'],['Nor'],['Nor'],['Nor'],['Nor'],['Nor'],['Nor'],['Nor']...]
decoded_preds[-10:]
(#10) [['Pre', 'RoT'],['Pre', 'SPC'],['Pre', 'RoT'],['Pre', 'SPC'],['Pre', 'SPC'],['Pre', 'RoT'],['Pre', 'SPC'],['Pre', 'RoT'],['Pre', 'RoT'],['Pre', 'SPC']]
tfms = [None, TSClassification()]
with tfms = [None, TSMultiLabelClassification()]
learn = ts_learner()
or learn = Learner()
ts_learner
you can pass an architecture (rather than passing pre-created model) or leave it as None, in which case the default (InceptiontimePlus) will be used.metrics=accuracy_multi
in place of metrics=accuracy
used in earlier multi-class examplettsai
will set it to loss function loss_func=BCEWithLogitsLossFlat()
.tsai
automatically calculates multi class weights.BCEWithLogitsLossFlat(pos_weight=dls.train.cws)
BCEWithLogitsLossFlat(pos_weight=dls.train.cws.sqrt())