%reload_ext autoreload
%autoreload 2
%matplotlib inline
from fastai import *
from fastai.vision import *
The planet dataset isn't available on the fastai dataset page due to copyright restrictions. You can download it from Kaggle however. Let's see how to do this by using the Kaggle API as it's going to be pretty useful to you if you want to join a competition or use other Kaggle datasets later on.
First, install the Kaggle API by uncommenting the following line and executing it, or by executing it in your terminal. Depending on your environment, you may need to append "--user" to the command.
# ! pip install kaggle --upgrade
Then you need to upload your credentials from Kaggle on your instance. Login to kaggle and click on your profile picture on the top left corner, then 'My account'. Scroll down until you find a button named 'Create New API Token' and click on it. This will trigger the download of a file named 'kaggle.json'.
Upload this file to the directory this notebook is running in, by clicking "Upload" on your main Jupyter page, then uncomment and execute the next two commands.
#! mkdir -p ~/.kaggle/
#! mv kaggle.json ~/.kaggle/
You're all set to download the data from planet competition. You just need to go to its main page and accept its rules, and run the two cells below (uncomment the shell commands to download and unzip the data).
path = Path(Config.get_key('data_path')).expanduser()/'planet'
os.makedirs(path, exist_ok=True)
path
PosixPath('/home/jhoward/.fastai/data/planet')
# ! kaggle competitions download -c planet-understanding-the-amazon-from-space -f train-jpg.tar.7z -p {path}
# ! kaggle competitions download -c planet-understanding-the-amazon-from-space -f train_v2.csv -p {path}
# ! unzip {path}/train_v2.csv.zip -d {path}
We only download the training data file for this lesson, but you can grab the test file if you want to make a late submission.
To extract the content of this file, we'll need 7zip, so uncomment the following line if you need to install it.
#! sudo apt-get install p7zip-full
And now we can finally unpack the data.
# ! 7za -bd -y x {path}/train-jpg.tar.7z -o{path}
# ! tar -xf {path}/train-jpg.tar -C {path}
path.ls()
[PosixPath('/home/jhoward/.fastai/data/planet/train_v2.csv'), PosixPath('/home/jhoward/.fastai/data/planet/models'), PosixPath('/home/jhoward/.fastai/data/planet/train-jpg.tar'), PosixPath('/home/jhoward/.fastai/data/planet/train_v2.csv.zip'), PosixPath('/home/jhoward/.fastai/data/planet/__MACOSX'), PosixPath('/home/jhoward/.fastai/data/planet/train-jpg.tar.7z'), PosixPath('/home/jhoward/.fastai/data/planet/train-jpg')]
Contrary to the pets dataset studied in last lesson, here each picture can have multiple labels. If we take a look at the csv file containing the labels (in 'train_v2.csv' here) we see that each 'image_name' is associated to several tags separated by spaces.
df = pd.read_csv(path/'train_v2.csv')
df.head()
image_name | tags | |
---|---|---|
0 | train_0 | haze primary |
1 | train_1 | agriculture clear primary water |
2 | train_2 | clear primary |
3 | train_3 | clear primary |
4 | train_4 | agriculture clear habitation primary road |
To put this in a DataBunch
while using the data block API, we then need to using ImageMultiDataset
(and not ImageClassificationDataset
). This will make sure the model created has the proper loss function to deal with the multiple classes.
tfms = get_transforms(flip_vert=True, max_lighting=0.1, max_zoom=1.05, max_warp=0.)
np.random.seed()
data = (ImageFileList.from_folder(path)
.label_from_csv('train_v2.csv', sep=' ', folder='train-jpg', suffix='.jpg')
.random_split_by_pct(0.2)
.datasets(ImageMultiDataset)
.transform(tfms, size=128)
.databunch()
.normalize(imagenet_stats))
show_batch
still works, and show us the different labels separated by ;.
data.show_batch(rows=3, figsize=(10,10))
To create a Learner
we use the same function as in lesson 1. Our base architecture is resnet34 again, but the metrics are a little bit differeent: we use accuracy_thresh
instead of accuracy
. In lesson 1, we determined the predicition for a given class by picking the final activation that was the biggest, but here, each activation can be 0. or 1. accuracy_thresh
selects the one that are above a certain threshold (0.5 by default) and comapre them to the ground truth.
As for Fbeta, it's the metric that was used by Kaggle on this competition. See here for more details.
f_score = partial(fbeta, thresh=0.2)
learn = create_cnn(data, models.resnet34, metrics=[accuracy_thresh, f_score])
We use the LR Finder to pick a good learning rate.
learn.lr_find()
epoch | train_loss | valid_loss | accuracy_thresh | fbeta |
---|---|---|---|---|
learn.recorder.plot()
Then we can fit the head of our network.
lr = 0.1
learn.fit_one_cycle(5, lr)
Total time: 02:18 epoch train_loss valid_loss accuracy_thresh fbeta 1 0.171888 0.148639 0.941067 0.870724 (00:28) 2 0.200026 0.448930 0.932108 0.833527 (00:27) 3 0.163548 17.766521 0.944737 0.867123 (00:27) 4 0.131419 0.116334 0.957752 0.903322 (00:27) 5 0.116350 0.102876 0.960578 0.912573 (00:27)
learn.unfreeze()
learn.fit_one_cycle(5, slice(lr/100, lr/5))
Total time: 03:10 epoch train_loss valid_loss accuracy_thresh fbeta 1 0.126622 20.821779 0.931490 0.862767 (00:38) 2 0.135166 0.168177 0.947753 0.877090 (00:37) 3 0.117995 0.135464 0.959583 0.908179 (00:38) 4 0.109174 0.094508 0.963892 0.918502 (00:38) 5 0.100283 0.098827 0.964517 0.920089 (00:38)