In [2]:
# %load /Users/facai/Study/book_notes/
%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sns
sns.set(font='SimHei', font_scale=2.5)
plt.rcParams['axes.grid'] = False

def show_image(filename, figsize=None, res_dir=True):
    if figsize:

    if res_dir:
        filename = './res/{}'.format(filename)


11 Practical Methodology

practical design process:

  1. Determine your goals.
    • which error metric to use?
      • coverage:
    • what level of performance you desire?
  2. Establish a working end-to-end pipeline as soon as possible.
  3. Instrument the system well to determine bottlenecks in performance.
  4. Repeatedly make incremental changes.
    • Determinig whether to gather more data <= when train error is accpted, while test error not.
      • plot curves showing the relationship between training set size and generalization error. usually logarithmic scale.
    • Selecting Hyperparameters
      • manually: learning rate
      • automatic
        1. grid search: typically use logarithmic scale
        2. random search: more convenient to use, and converges much faster
    • Debugging strategies
      1. Visualize the model in action.
      2. Visualize the worst mistakes.
      3. Reasoning about software using train and test error.
      4. Fit a tiny dataset.
      5. Compare back-propagated derivatives to numerical derivatives.
      6. Monitor histograms of activations and gradient:
        • visualize statistics of neural network activations and gradients, collected over a large amount of training iterations (maybe one epoch).
        • compare the magnitude of parameter gradients to themselves.