MURA | Abnormality detection

What is MURA?

MURA (musculoskeletal radiographs) is a large dataset of bone X-rays. Algorithms are tasked with determining whether an X-ray study is normal or abnormal.

Musculoskeletal conditions affect more than 1.7 billion people worldwide, and are the most common cause of severe, long-term pain and disability, with 30 million emergency department visits annually and increasing. We hope that our dataset can lead to significant advances in medical imaging technologies which can diagnose at the level of experts, towards improving healthcare access in parts of the world where access to skilled radiologists is limited.

MURA is one of the largest public radiographic image datasets. We're making this dataset available to the community and hosting a competition to see if your models can perform as well as radiologists on the task.

Initialisation

In [1]:
%reload_ext autoreload
%autoreload 2
%matplotlib inline
In [2]:
from fastai.vision import *
from fastai.widgets import *
import shutil
In [3]:
from fastai.callbacks import * 
In [4]:
import fastai
print(f'fastai: {fastai.__version__}')
print(f'cuda: {torch.cuda.is_available()}')
fastai: 1.0.48
cuda: True
In [5]:
import gc
import torch

Data

Data path

In [6]:
Config.data_path()
Out[6]:
PosixPath('/home/jupyter/tutorials/data')
In [7]:
path = Config.data_path() / 'MURA-v1.1'
path.ls()
Out[7]:
[PosixPath('/home/jupyter/tutorials/data/MURA-v1.1/valid_labeled_studies.csv'),
 PosixPath('/home/jupyter/tutorials/data/MURA-v1.1/valid'),
 PosixPath('/home/jupyter/tutorials/data/MURA-v1.1/train_image_paths.csv'),
 PosixPath('/home/jupyter/tutorials/data/MURA-v1.1/train_labeled_studies.csv'),
 PosixPath('/home/jupyter/tutorials/data/MURA-v1.1/valid_image_paths.csv'),
 PosixPath('/home/jupyter/tutorials/data/MURA-v1.1/train'),
 PosixPath('/home/jupyter/tutorials/data/MURA-v1.1/data2')]

csv

In [8]:
df_train = pd.read_csv(path / 'train_image_paths.csv', header=None, names=['image'])
df_train.image[0]
Out[8]:
'MURA-v1.1/train/XR_SHOULDER/patient00001/study1_positive/image1.png'
In [9]:
df_valid = pd.read_csv(path / 'valid_image_paths.csv', header=None, names=['image'])
df_valid.image[0]
Out[9]:
'MURA-v1.1/valid/XR_WRIST/patient11185/study1_positive/image1.png'
In [10]:
df_train_label = pd.read_csv(path / 'train_labeled_studies.csv', header=None, names=['image', 'label'])
df_train_label.head()
Out[10]:
image label
0 MURA-v1.1/train/XR_SHOULDER/patient00001/study... 1
1 MURA-v1.1/train/XR_SHOULDER/patient00002/study... 1
2 MURA-v1.1/train/XR_SHOULDER/patient00003/study... 1
3 MURA-v1.1/train/XR_SHOULDER/patient00004/study... 1
4 MURA-v1.1/train/XR_SHOULDER/patient00005/study... 1

Create folder data2 to train models

In [11]:
# Create data2 with data to train our models
path_train = path / 'data2/train'
path_valid = path / 'data2/valid'
path_train.mkdir(parents=True, exist_ok=True)
path_valid.mkdir(parents=True, exist_ok=True)
In [12]:
path_train_neg = path_train / '0'
path_train_pos = path_train / '1'
path_train_neg.mkdir(parents=True, exist_ok=True)
path_train_pos.mkdir(parents=True, exist_ok=True)

path_valid_neg = path_valid / '0'
path_valid_pos = path_valid / '1'
path_valid_neg.mkdir(parents=True, exist_ok=True)
path_valid_pos.mkdir(parents=True, exist_ok=True)

Get list of images

In [13]:
fnames_train = get_image_files(path/'train', recurse=True)
print(len(fnames_train))
fnames_train[:5]
36808
Out[13]:
[PosixPath('/home/jupyter/tutorials/data/MURA-v1.1/train/XR_FINGER/patient04021/study1_negative/image1.png'),
 PosixPath('/home/jupyter/tutorials/data/MURA-v1.1/train/XR_FINGER/patient03816/study1_negative/image1.png'),
 PosixPath('/home/jupyter/tutorials/data/MURA-v1.1/train/XR_FINGER/patient04345/study1_negative/image2.png'),
 PosixPath('/home/jupyter/tutorials/data/MURA-v1.1/train/XR_FINGER/patient04345/study1_negative/image3.png'),
 PosixPath('/home/jupyter/tutorials/data/MURA-v1.1/train/XR_FINGER/patient04345/study1_negative/image1.png')]
In [14]:
fnames_valid = get_image_files(path/'valid', recurse=True)
print(len(fnames_valid))
fnames_valid[:5]
3197
Out[14]:
[PosixPath('/home/jupyter/tutorials/data/MURA-v1.1/valid/XR_FINGER/patient11855/study1_negative/image1.png'),
 PosixPath('/home/jupyter/tutorials/data/MURA-v1.1/valid/XR_FINGER/patient11606/study1_negative/image1.png'),
 PosixPath('/home/jupyter/tutorials/data/MURA-v1.1/valid/XR_FINGER/patient11513/study1_negative/image1.png'),
 PosixPath('/home/jupyter/tutorials/data/MURA-v1.1/valid/XR_FINGER/patient11911/study1_positive/image2.png'),
 PosixPath('/home/jupyter/tutorials/data/MURA-v1.1/valid/XR_FINGER/patient11911/study1_positive/image3.png')]

Copy images into data2

In [15]:
pat_label = re.compile(r'/XR_([^/]+)/[^/]+/[^/]+/[^/]+.png$')
pat_patient = re.compile(r'/[^/]+/patient([^/]+)/[^/]+/[^/]+.png$')
pat_study = re.compile(r'/[^/]+/[^/]+/([^/]+)/[^/]+.png$')
# pat_study_negpos = re.compile(r'\\[^\\]+\\[^\\]+\\study\d+_([^\\]+)\\[^\\]+.png$')
In [39]:
%%time
# copy all train images in corresponding class folders under MURA-v1.1/data2/train
for src in fnames_train:
    # get image label
    label = pat_label.search(str(src))
    label = label.group(1)
    # get patient number
    patient = pat_patient.search(str(src))
    patient = patient.group(1)
    # get study name
    study = pat_study.search(str(src))
    study = study.group(1)  
    # create class folder if necessary
    if 'negative' in study:
        path_label = path_train_neg
    else:
        path_label = path_train_pos
    # copy image to its class folder
    img_name = label + '_patient' + patient + '_' + study + '_' + src.name
    dest = path_label / img_name
    shutil.copy(str(src), str(dest))
CPU times: user 3.42 s, sys: 6.41 s, total: 9.83 s
Wall time: 2min 22s
In [40]:
%%time
# copy all valid images in corresponding class folders under MURA-v1.1/data2/valid
for src in fnames_valid:
    # get image label
    label = pat_label.search(str(src))
    label = label.group(1)
    # get patient number
    patient = pat_patient.search(str(src))
    patient = patient.group(1)
    # get study name
    study = pat_study.search(str(src))
    study = study.group(1)  
    # create class folder if necessary
    if 'negative' in study:
        path_label = path_valid_neg
    else:
        path_label = path_valid_pos
    # copy image to its class folder
    img_name = label + '_patient' + patient + '_' + study + '_' + src.name
    dest = path_label / img_name
    shutil.copy(str(src), str(dest))
CPU times: user 336 ms, sys: 564 ms, total: 900 ms
Wall time: 11.4 s

Number of studies

In [16]:
pat_label = re.compile(r'/XR_([^/]+)/[^/]+/[^/]+/[^/]+.png$')
pat_patient = re.compile(r'/[^/]+/patient([^/]+)/[^/]+/[^/]+.png$')
pat_study = re.compile(r'/([^/]+)_[^/]+/[^/]+.png$')
In [17]:
mura = ['elbow', 'finger', 'forearm', 'hand', 'humerus', 'shoulder', 'wrist']

study_train_dict = dict()
study_valid_dict = dict()

for m in mura:
    study_train_dict[m] = list()
    study_valid_dict[m] = list()
    
for src in fnames_train:
    # get image label
    label = pat_label.search(str(src))
    label = label.group(1)
    # get patient number
    patient = pat_patient.search(str(src))
    patient = patient.group(1)
    # get study name
    study = pat_study.search(str(src))
    study = study.group(1)
    # add to label list
    s = 'patient' + patient + '_' + study
    study_train_dict[label.lower()].append(s)

for src in fnames_valid:
    # get image label
    label = pat_label.search(str(src))
    label = label.group(1)
    # get patient number
    patient = pat_patient.search(str(src))
    patient = patient.group(1)
    # get study name
    study = pat_study.search(str(src))
    study = study.group(1)
    # add to label list
    s = 'patient' + patient + '_' + study
    study_valid_dict[label.lower()].append(s)
In [18]:
num_train_studies = 0
num_valid_studies = 0

for m in mura:
    # train
    myset = set(study_train_dict[m])
    num_train_studies += len(myset)
    # valid
    myset = set(study_valid_dict[m])
    num_valid_studies += len(myset)
In [19]:
# 207 studies in test
num_train_studies, num_valid_studies, num_train_studies + num_valid_studies + 207
Out[19]:
(13457, 1199, 14863)

Training with resnet34

size = 112

In [93]:
size = 112
bs = 512

np.random.seed(42)
data = ImageDataBunch.from_folder(path/'data2', ds_tfms=get_transforms(flip_vert=True, max_warp=0.), 
                                  size=size, bs=bs, 
                                  ).normalize(imagenet_stats)
In [19]:
data.show_batch(rows=3, figsize=(7,6))
In [20]:
data.classes
Out[20]:
['0', '1']
In [21]:
len(data.train_ds), len(data.valid_ds), len(data.train_ds) + len(data.valid_ds)
Out[21]:
(36808, 3197, 40005)
In [22]:
plt.bar([0,1], [len(path_train_neg.ls()), len(path_train_pos.ls())])
plt.show()
In [94]:
learn = cnn_learner(data, models.resnet34, metrics=[error_rate, accuracy], wd=0.1)
In [17]:
learn.fit_one_cycle(5,callbacks=[ShowGraph(learn),SaveModelCallback(learn)])
Total time: 1:10:49

epoch train_loss valid_loss error_rate accuracy time
1 0.777486 0.657900 0.380044 0.619956 20:09
2 0.656167 0.632573 0.353769 0.646231 12:55
3 0.614780 0.602607 0.324679 0.675321 12:23
4 0.592553 0.595166 0.322803 0.677197 12:34
5 0.585395 0.598298 0.325618 0.674382 12:44
Better model found at epoch 1 with val_loss value: 0.6579000353813171.
Better model found at epoch 2 with val_loss value: 0.6325731873512268.
Better model found at epoch 3 with val_loss value: 0.6026073694229126.
Better model found at epoch 4 with val_loss value: 0.5951657891273499.
In [39]:
learn.fit_one_cycle(5,callbacks=[ShowGraph(learn),SaveModelCallback(learn)])
Total time: 06:55

epoch train_loss valid_loss error_rate accuracy time
0 0.767645 0.673345 0.381608 0.618392 01:20
1 0.653763 0.621917 0.346575 0.653425 01:22
2 0.612328 0.622118 0.341883 0.658117 01:23
3 0.590675 0.595303 0.323741 0.676259 01:24
4 0.581991 0.597755 0.322803 0.677197 01:21
Better model found at epoch 0 with val_loss value: 0.6733449101448059.
Better model found at epoch 1 with val_loss value: 0.6219168901443481.
Better model found at epoch 3 with val_loss value: 0.5953028202056885.
In [40]:
learn.save('resnet34-stage-1')
In [21]:
learn.load('resnet34-stage-1');
In [41]:
learn.purge()
learn.unfreeze()
In [42]:
learn.lr_find()
LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.
In [43]:
learn.recorder.plot()
In [25]:
lr=1e-4
learn.fit_one_cycle(5,max_lr=slice(lr/100,lr),callbacks=[ShowGraph(learn),SaveModelCallback(learn)])
Total time: 1:19:50

epoch train_loss valid_loss error_rate accuracy time
1 0.578047 0.588865 0.314670 0.685330 14:43
2 0.566004 0.573437 0.297466 0.702534 16:19
3 0.552305 0.558046 0.283703 0.716297 16:15
4 0.542241 0.556961 0.284016 0.715984 16:14
5 0.537519 0.553843 0.283703 0.716297 16:13
Better model found at epoch 1 with val_loss value: 0.5888653993606567.
Better model found at epoch 2 with val_loss value: 0.5734368562698364.
Better model found at epoch 3 with val_loss value: 0.5580459237098694.
Better model found at epoch 4 with val_loss value: 0.5569610595703125.
Better model found at epoch 5 with val_loss value: 0.5538430213928223.
In [44]:
lr=1e-4
learn.fit_one_cycle(5,max_lr=slice(lr/100,lr),callbacks=[ShowGraph(learn),SaveModelCallback(learn)])
Total time: 07:13

epoch train_loss valid_loss error_rate accuracy time
0 0.576275 0.590239 0.319049 0.680951 01:24
1 0.564980 0.572742 0.298405 0.701595 01:25
2 0.551877 0.564305 0.291211 0.708789 01:25
3 0.543682 0.561075 0.286831 0.713169 01:26
4 0.537840 0.559501 0.285580 0.714420 01:25
Better model found at epoch 0 with val_loss value: 0.5902385711669922.
Better model found at epoch 1 with val_loss value: 0.5727419257164001.
Better model found at epoch 2 with val_loss value: 0.5643052458763123.
Better model found at epoch 3 with val_loss value: 0.5610751509666443.
Better model found at epoch 4 with val_loss value: 0.5595011115074158.
In [45]:
learn.save('resnet34-stage-2')

Results by image

In [95]:
size = 112
bs = 512

np.random.seed(42)
data = ImageDataBunch.from_folder(path/'data2', ds_tfms=get_transforms(flip_vert=True, max_warp=0.), 
                                  size=size, bs=bs, 
                                  ).normalize(imagenet_stats)
In [96]:
learn = cnn_learner(data, models.resnet34, metrics=[error_rate, accuracy], wd=0.1)
In [97]:
learn.load('resnet34-stage-2');
In [98]:
interp = ClassificationInterpretation.from_learner(learn)

losses,idxs = interp.top_losses()

len(data.valid_ds)==len(losses)==len(idxs)
Out[98]:
True
In [99]:
# first interpretation
interp.plot_confusion_matrix()
In [100]:
interp.most_confused(min_val=2)
Out[100]:
[('1', '0', 723), ('0', '1', 184)]
In [101]:
interp.plot_top_losses(9, figsize=(15,11))

Results by study

In [102]:
learn.load('resnet34-stage-2');
In [103]:
# validation
preds_val, y_val = learn.get_preds()
In [104]:
preds_val
Out[104]:
tensor([[0.2962, 0.7038],
        [0.1308, 0.8692],
        [0.4768, 0.5232],
        ...,
        [0.7501, 0.2499],
        [0.8186, 0.1814],
        [0.6868, 0.3132]])
In [105]:
len(preds_val)
Out[105]:
3197
In [106]:
for img_url in data.valid_ds.x.items:
    print(img_url)
    break
/home/jupyter/tutorials/data/MURA-v1.1/data2/valid/1/HAND_patient11538_study1_positive_image1.png
In [107]:
pat_label = re.compile(r'/([^/]+)_patient[^/]+.png$')
pat_study = re.compile(r'/([^/]+)_[^_]+.png$')
In [108]:
%%time
studies = dict()
studies_num = dict()
labels_num = dict()

for m in mura:
    labels_num[m] = 0

for idx, src in enumerate(data.valid_ds.x.items):
    # get label name
    label = pat_label.search(str(src))
    label = label.group(1) 
    # get study name
    study = pat_study.search(str(src))
    study = study.group(1) 
    # sum probabilities by study
    if study in studies:
        studies[study] += preds_val[idx,:].clone()
        studies_num[study] += 1
    else:
        studies[study] = preds_val[idx,:].clone()
        studies_num[study] = 1
    labels[study] = label
CPU times: user 192 ms, sys: 0 ns, total: 192 ms
Wall time: 50.2 ms
In [110]:
labels_num = dict()
for m in mura:
    labels_num[m] = sum([1 for k,v in labels.items() if v.lower() == m])
In [111]:
print(labels_num)
print(sum([v for k,v in labels_num.items()]))
{'elbow': 158, 'finger': 175, 'forearm': 133, 'hand': 167, 'humerus': 135, 'shoulder': 194, 'wrist': 237}
1199
In [113]:
len(studies)
Out[113]:
1199
In [115]:
len(studies_num)
Out[115]:
1199
In [116]:
# get averages
for (k,v) in studies.items():
    studies[k] = studies[k] / studies_num[k]
In [118]:
# get predictions by study
acc = 0.
for (k,v) in studies.items():
    prob, y_hat = torch.max(studies[k],0)
    if 'negative' in k:
        acc += (0 == y_hat.item())
    else:
        acc += (1 == y_hat.item())
    # print(f'{k} {y_hat.item()} ({prob})')
In [119]:
len(studies), acc
Out[119]:
(1199, 876.0)
In [120]:
# get study accuracy total
print(f'study accuracy total: {round(acc / len(studies),3)}')
study accuracy total: 0.731
In [121]:
# get predictions by study and label
acc_label = dict()
for m in mura:
    acc_label[m] = 0
    
for (k,v) in studies.items():
    prob, y_hat = torch.max(studies[k],0)
    label = labels[k]
    if 'negative' in k:
        acc_label[label.lower()] += (0 == y_hat.item())
    else:
        acc_label[label.lower()] += (1 == y_hat.item())
In [122]:
acc_label
Out[122]:
{'elbow': 120,
 'finger': 121,
 'forearm': 105,
 'hand': 112,
 'humerus': 102,
 'shoulder': 130,
 'wrist': 186}
In [123]:
sum([v for k,v in acc_label.items()])
Out[123]:
876
In [124]:
labels_num
Out[124]:
{'elbow': 158,
 'finger': 175,
 'forearm': 133,
 'hand': 167,
 'humerus': 135,
 'shoulder': 194,
 'wrist': 237}
In [125]:
# get study accuracy by label
for m in mura:
    print(f'{m}: {round(acc_label[m] / labels_num[m],3)}')
elbow: 0.759
finger: 0.691
forearm: 0.789
hand: 0.671
humerus: 0.756
shoulder: 0.67
wrist: 0.785

size = 224

In [149]:
learn = None
gc.collect()
Out[149]:
7046
In [150]:
torch.cuda.empty_cache()
In [151]:
size = 224
bs = 128

np.random.seed(42)
data = ImageDataBunch.from_folder(path/'data2', ds_tfms=get_transforms(flip_vert=True, max_warp=0.), 
                                  size=size, bs=bs, 
                                  ).normalize(imagenet_stats)
In [152]:
learn = cnn_learner(data, models.resnet34, metrics=[error_rate, accuracy], wd=0.1)
In [153]:
learn.load('resnet34-stage-2');
In [154]:
learn.fit_one_cycle(5,callbacks=[ShowGraph(learn),SaveModelCallback(learn)])
Total time: 13:40

epoch train_loss valid_loss error_rate accuracy time
0 0.543209 0.529940 0.258993 0.741007 02:43
1 0.511583 0.511662 0.243666 0.756334 02:43
2 0.497246 0.504433 0.238661 0.761339 02:43
3 0.480150 0.498858 0.236472 0.763528 02:43
4 0.473954 0.486756 0.229590 0.770410 02:43
Better model found at epoch 0 with val_loss value: 0.5299397706985474.
Better model found at epoch 1 with val_loss value: 0.5116621851921082.
Better model found at epoch 2 with val_loss value: 0.5044328570365906.
Better model found at epoch 3 with val_loss value: 0.4988577663898468.
Better model found at epoch 4 with val_loss value: 0.4867563545703888.
In [155]:
learn.save('resnet34-stage-3');
In [156]:
learn = None
gc.collect()
torch.cuda.empty_cache()
In [157]:
learn = cnn_learner(data, models.resnet34, metrics=[error_rate, accuracy], wd=0.1)
learn.load('resnet34-stage-3');
In [158]:
learn.lr_find()
LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.
In [159]:
learn.recorder.plot()
In [160]:
lr = 3e-4
learn.fit_one_cycle(5,max_lr=lr,callbacks=[ShowGraph(learn),SaveModelCallback(learn)])
Total time: 13:36

epoch train_loss valid_loss error_rate accuracy time
0 0.480293 0.487448 0.228026 0.771974 02:41
1 0.474168 0.482256 0.222709 0.777291 02:42
2 0.468879 0.483917 0.227401 0.772599 02:44
3 0.461243 0.477690 0.223334 0.776666 02:43
4 0.468197 0.475864 0.221145 0.778855 02:43
Better model found at epoch 0 with val_loss value: 0.4874477982521057.
Better model found at epoch 1 with val_loss value: 0.4822562336921692.
Better model found at epoch 3 with val_loss value: 0.47769007086753845.
Better model found at epoch 4 with val_loss value: 0.47586414217948914.
In [161]:
learn.save('resnet34-stage-4')
In [36]:
learn.load('resnet34-stage-4');
In [162]:
learn.purge()
learn.unfreeze()
In [163]:
learn.lr_find()
LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.
In [164]:
learn.recorder.plot()
In [165]:
lr=3e-6
learn.fit_one_cycle(5,max_lr=slice(lr/100,lr),callbacks=[ShowGraph(learn),SaveModelCallback(learn)])
Total time: 18:17

epoch train_loss valid_loss error_rate accuracy time
0 0.467906 0.474374 0.221458 0.778542 03:37
1 0.464342 0.472145 0.219581 0.780419 03:39
2 0.466083 0.478271 0.222709 0.777291 03:38
3 0.460885 0.479489 0.223960 0.776040 03:39
4 0.460538 0.474725 0.221770 0.778230 03:39
Better model found at epoch 0 with val_loss value: 0.47437402606010437.
Better model found at epoch 1 with val_loss value: 0.47214457392692566.
In [166]:
learn.save('resnet34-stage-5')

Results by image

In [167]:
size = 224
bs = 128

np.random.seed(42)
data = ImageDataBunch.from_folder(path/'data2', ds_tfms=get_transforms(flip_vert=True, max_warp=0.), 
                                  size=size, bs=bs, 
                                  ).normalize(imagenet_stats)
In [168]:
learn = cnn_learner(data, models.resnet34, metrics=[error_rate, accuracy], wd=0.1)
In [169]:
learn.load('resnet34-stage-5');
In [170]:
interp = ClassificationInterpretation.from_learner(learn)

losses,idxs = interp.top_losses()

len(data.valid_ds)==len(losses)==len(idxs)
Out[170]:
True
In [171]:
# first interpretation
interp.plot_confusion_matrix()
In [172]:
interp.most_confused(min_val=2)
Out[172]:
[('1', '0', 531), ('0', '1', 171)]
In [173]:
interp.plot_top_losses(9, figsize=(15,11))

Results by study

In [174]:
learn.load('resnet34-stage-5');
In [175]:
# validation
preds_val, y_val = learn.get_preds()
In [176]:
preds_val
Out[176]:
tensor([[0.0743, 0.9257],
        [0.0709, 0.9291],
        [0.1770, 0.8230],
        ...,
        [0.8040, 0.1960],
        [0.8942, 0.1058],
        [0.6382, 0.3618]])
In [177]:
len(preds_val)
Out[177]:
3197
In [178]:
for img_url in data.valid_ds.x.items:
    print(img_url)
    break
/home/jupyter/tutorials/data/MURA-v1.1/data2/valid/1/HAND_patient11538_study1_positive_image1.png
In [179]:
pat_label = re.compile(r'/([^/]+)_patient[^/]+.png$')
pat_study = re.compile(r'/([^/]+)_[^_]+.png$')
In [180]:
%%time
studies = dict()
studies_num = dict()
labels_num = dict()

for m in mura:
    labels_num[m] = 0

for idx, src in enumerate(data.valid_ds.x.items):
    # get label name
    label = pat_label.search(str(src))
    label = label.group(1) 
    # get study name
    study = pat_study.search(str(src))
    study = study.group(1) 
    # sum probabilities by study
    if study in studies:
        studies[study] += preds_val[idx,:].clone()
        studies_num[study] += 1
    else:
        studies[study] = preds_val[idx,:].clone()
        studies_num[study] = 1
    labels[study] = label
CPU times: user 208 ms, sys: 0 ns, total: 208 ms
Wall time: 49.5 ms
In [182]:
labels_num = dict()
for m in mura:
    labels_num[m] = sum([1 for k,v in labels.items() if v.lower() == m])
In [183]:
print(labels_num)
print(sum([v for k,v in labels_num.items()]))
{'elbow': 158, 'finger': 175, 'forearm': 133, 'hand': 167, 'humerus': 135, 'shoulder': 194, 'wrist': 237}
1199
In [185]:
len(studies)
Out[185]:
1199
In [187]:
len(studies_num)
Out[187]:
1199
In [188]:
# get averages
for (k,v) in studies.items():
    studies[k] = studies[k] / studies_num[k]
In [190]:
# get predictions by study
acc = 0.
for (k,v) in studies.items():
    prob, y_hat = torch.max(studies[k],0)
    if 'negative' in k:
        acc += (0 == y_hat.item())
    else:
        acc += (1 == y_hat.item())
    # print(f'{k} {y_hat.item()} ({prob})')
In [191]:
len(studies), acc
Out[191]:
(1199, 955.0)
In [192]:
# get study accuracy total
print(f'study accuracy total: {round(acc / len(studies),3)}')
study accuracy total: 0.796
In [193]:
# get predictions by study and label
acc_label = dict()
for m in mura:
    acc_label[m] = 0
    
for (k,v) in studies.items():
    prob, y_hat = torch.max(studies[k],0)
    label = labels[k]
    if 'negative' in k:
        acc_label[label.lower()] += (0 == y_hat.item())
    else:
        acc_label[label.lower()] += (1 == y_hat.item())
In [194]:
acc_label
Out[194]:
{'elbow': 129,
 'finger': 132,
 'forearm': 113,
 'hand': 124,
 'humerus': 110,
 'shoulder': 148,
 'wrist': 199}
In [195]:
sum([v for k,v in acc_label.items()])
Out[195]:
955
In [196]:
labels_num
Out[196]:
{'elbow': 158,
 'finger': 175,
 'forearm': 133,
 'hand': 167,
 'humerus': 135,
 'shoulder': 194,
 'wrist': 237}
In [197]:
# get study accuracy by label
for m in mura:
    print(f'{m}: {round(acc_label[m] / labels_num[m],3)}')
elbow: 0.816
finger: 0.754
forearm: 0.85
hand: 0.743
humerus: 0.815
shoulder: 0.763
wrist: 0.84

Training with densenet169

size = 112

In [29]:
import gc
import torch
In [30]:
learn = None
gc.collect()
Out[30]:
16898
In [31]:
torch.cuda.empty_cache()
In [62]:
size = 112
bs = 256

np.random.seed(42)
data = ImageDataBunch.from_folder(path/'data2', ds_tfms=get_transforms(flip_vert=True, max_warp=0.), 
                                  size=size, bs=bs, 
                                  ).normalize(imagenet_stats)
In [63]:
learn = cnn_learner(data, models.densenet169, metrics=[error_rate, accuracy], wd=0.1)
learn = learn.to_fp16()
In [64]:
learn.fit_one_cycle(5,callbacks=[ShowGraph(learn),SaveModelCallback(learn)])
Total time: 08:56

epoch train_loss valid_loss error_rate accuracy time
0 0.647629 0.605026 0.325931 0.674069 01:56
1 0.553631 0.549180 0.288395 0.711605 01:45
2 0.523585 0.512760 0.249922 0.750078 01:44
3 0.503554 0.505001 0.247732 0.752268 01:44
4 0.486226 0.510447 0.249922 0.750078 01:44
Better model found at epoch 0 with val_loss value: 0.6050257086753845.
Better model found at epoch 1 with val_loss value: 0.549180269241333.
Better model found at epoch 2 with val_loss value: 0.5127597451210022.
Better model found at epoch 3 with val_loss value: 0.5050009489059448.
In [65]:
learn.save('densenet169-stage-1')
In [66]:
learn.load('densenet169-stage-1');
learn = learn.to_fp16()
In [67]:
learn.purge()
learn.unfreeze()
In [68]:
learn.lr_find()
LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.
In [69]:
learn.recorder.plot()
In [70]:
lr=1e-4
learn.fit_one_cycle(5,max_lr=slice(lr/100,lr),callbacks=[ShowGraph(learn),SaveModelCallback(learn)])
Total time: 10:39

epoch train_loss valid_loss error_rate accuracy time
0 0.492801 0.509401 0.248983 0.751017 02:07
1 0.487105 0.498616 0.239600 0.760400 02:07
2 0.485631 0.496530 0.237723 0.762277 02:07
3 0.476899 0.494938 0.233969 0.766031 02:07
4 0.476169 0.492549 0.233344 0.766656 02:06
Better model found at epoch 0 with val_loss value: 0.5094006061553955.
Better model found at epoch 1 with val_loss value: 0.4986157715320587.
Better model found at epoch 2 with val_loss value: 0.4965299963951111.
Better model found at epoch 3 with val_loss value: 0.49493834376335144.
Better model found at epoch 4 with val_loss value: 0.49254900217056274.
In [71]:
learn.save('densenet169-stage-2')

Results by image

In [72]:
size = 112
bs = 256

np.random.seed(42)
data = ImageDataBunch.from_folder(path/'data2', ds_tfms=get_transforms(flip_vert=True, max_warp=0.), 
                                  size=size, bs=bs, 
                                  ).normalize(imagenet_stats)
In [73]:
learn = cnn_learner(data, models.densenet169, metrics=[error_rate, accuracy], wd=0.1)
In [74]:
learn.load('densenet169-stage-2');
In [75]:
interp = ClassificationInterpretation.from_learner(learn)

losses,idxs = interp.top_losses()

len(data.valid_ds)==len(losses)==len(idxs)
Out[75]:
True
In [76]:
# first interpretation
interp.plot_confusion_matrix()
In [77]:
interp.most_confused(min_val=2)
Out[77]:
[('1', '0', 572), ('0', '1', 171)]
In [78]:
interp.plot_top_losses(9, figsize=(15,11))