import xview
from os import path
# PLEASE EDIT THE FOLLOWING PATHS FOR YOUR LOCAL SETTINGS
# path where the image data will be/is stored
DATA_BASEPATH = '/tmp/test'
# path where experiment configs will be stored
EXPERIMENT_STORAGE_FOLDER = '/tmp/exp'
# only change in case you want to use tensorboard for model training, otherwise not relevant
EXP_OUT = '/tmp'
print('writing settings to %s' % path.join(path.dirname(xview.__file__), 'settings.py'))
with open(path.join(path.dirname(xview.__file__), 'settings.py'), 'w') as settings_file:
settings_file.write("DATA_BASEPATH = '%s'\n" % DATA_BASEPATH)
settings_file.write("EXPERIMENT_STORAGE_FOLDER = '%s'\n" % EXPERIMENT_STORAGE_FOLDER)
settings_file.write("EXP_OUT = '%s'\n" % EXP_OUT)
2. Downloading Experimental Data
All training and measurement experiments are bundeled into an archive that is downloaded and installed. This gives you access to pre-trained models and all experimental configurations.
! wget http://robotics.ethz.ch/~asl-datasets/2018_modular_semantic_segmentation/experimental_data.tar.gz -O /tmp/experimental_data.tar.gz
import tarfile
from os import path
from xview.settings import EXPERIMENT_STORAGE_FOLDER
tar = tarfile.open('/tmp/experimental_data.tar.gz')
tar.extractall(path=EXPERIMENT_STORAGE_FOLDER)
tar.close()
! rm /tmp/experimental_data.tar.gz
The following cells load the results from our timing experiments.
The inference time was measured using on a single GPU that did not have enough memory to make use of any parallellisation possibilities. In addition, the inference time measurements do not include time required to load data into the gpu as this heavily depends on code optimization and used hardware specs. We evaluated the inference time on a defined constant of equal size as the actual RGB and Depth images used. For more details, please have a look at the implementation of the experiment.
from experiments.utils import ExperimentData
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
# Load the relevant experiments. If this fails, your config from above is not setup properly.
rgb = ExperimentData(1062)
depth = ExperimentData(1063)
bayes = ExperimentData(1071)
dirichlet = ExperimentData(1067)
variance = ExperimentData(1065)
fusionfcn = ExperimentData(1059)
average = ExperimentData(1064)
def print_table(**experiments):
table_data = {}
for key, experiment_id in experiments.items():
text = ExperimentData(experiment_id).get_record()['captured_out'].split('\n')
line = next(line for line in text if line.startswith('Mean Time'))
mean = float(line[10:17])
std = float(line[24:31])
table_data[key] = [mean, std]
classes = pd.DataFrame(table_data, index=['mean', 'std'])
# now order the columns
ordered_columns = []
for key in ['Fusion_Fcn', 'Dirichlet', 'Bayes', 'Average', 'Variance', 'RGB', 'Depth']:
if key in classes.columns:
ordered_columns.append(key)
return classes[ordered_columns]
pd.set_option('display.precision', 4)
print_table(RGB=1062, Depth=1063, Bayes=1071, Dirichlet=1067, Variance=1065, Fusion_Fcn=1059, Average=1064)
Fusion_Fcn | Dirichlet | Bayes | Average | Variance | RGB | Depth | |
---|---|---|---|---|---|---|---|
mean | 0.0720 | 0.0517 | 0.0461 | 0.0432 | 0.3064 | 0.0219 | 0.0218 |
std | 0.0221 | 0.0238 | 0.0156 | 0.0113 | 0.0183 | 0.0114 | 0.0121 |
To reproduce the evaluation experiments listed above, simply invoke the following command with the correct experiment_id
:
More info on the experiment management can be found here.
The timing results in the paper were measured on an NVIDIA GTX 1080 Ti major. Any other GPU or CPU, aswell as other hardware setups can have influence on the timing analysis. It is therefore very hard to exactly reproduce the above results.
! python -m experiments.rerun with experiment_id=1071 -u