Homework 2

In [1]:
__author__ = "Christopher Potts"
__version__ = "CS224u, Stanford, Spring 2018 term"

This homework covers material from the sentiment classification unit. The primary value of doing the work is that it provides more hands-on experience with the dataset and the models we explored. All the code you write has potential value in the bake-off for this unit as well.

Submission URL: https://canvas.stanford.edu/courses/83399/quizzes/50657

Questions 1–4: Reproducing a Socher et al's NaiveBayes baseline [4 points]

Socher et al. 2013 compare against a Naive Bayes baseline with bigram features. See how close you can come to reproducing the performance of this model on the binary, root-only problem (values in the rightmost column of their Table 1, rows 1 and 3).

Specific tasks:

  1. Write a bigrams feature function, on the model of unigrams_phi. Call this bigrams_phi. In writing this function, ensure that each example is padded with start and end symbols (say, <S> and </S>), so that these contexts are properly reflected in the feature space.

  2. Write a function fit_nb_classifier_with_crossvalidation that serves as a wrapper for sklearn.naive_bayes.MultinomialNB and searches over these values for the smoothing parameter alpha: [0.1, 0.5, 1.0, 2.0], using 3-fold cross-validation.

  3. Use sst.experiment to run the experiments, assessing against dev_reader.

To submit:

  1. Your bigrams_phi
  2. Your fit_nb_classifier
  3. Your call to sst.experiment
  4. The average F1 score that sst.experiment reported.

A note on performance: in our experience, the bigrams Naive Bayes model gets around 0.75. It's fine to submit answers with comparable numbers; the Socher et al. baselines are very strong. We're not evaluating how good your model is; we want to see your code, and we're interested to see what the range of F1 scores is across the whole class.

Question 5–6: A more powerful vector-summing baseline [4 points]

In the section Distributed representations as features, we looked at a baseline for the binary SST problem in which each example is modeled as the sum of its 50-dimensional GloVe representations. A LogisticRegression model was used for prediction. A neural network might do better here, since there might be complex relationships between the input feature dimensions that a linear classifier can't learn.

To address this question, rerun the experiment with tf_shallow_neural_classifier.TfShallowNeuralClassifier as the classifier. Specs:

  • Use sst.experiment to conduct the experiment.
  • Using 3-fold cross-validation, exhaustively explore this set of hyperparameter combinations:
    • The hidden dimensionality at 50, 100, and 200.
    • The hidden activation function as tf.nn.tanh or tf.nn.relu.
  • (For all other parameters to TfShallowNeuralClassifier, use the defaults.)

To submit:

  • Your average F1 score according to sst.experiment.
  • The optimal hyperparameters chosen in your experiment. (You can just paste in the dict that sst._experiment prints.)

No need to include your supporting code.

We're not evaluating the quality of your model. (We've specified the protocols completely, but there will still be a lot of variation in the results.) However, the primary goal of this question is to get you thinking more about this strikingly good baseline feature representation scheme for SST, so we're sort of hoping you feel compelled to try out variations on your own.

Questions 7–8: Bidirectional RNN [2 points]

The auxiliary notebook tensorflow_models.ipynb subclasses TfRNNClassifier with a bidirectional RNN. In this model, the RNN is run in both directions and the concatenation of the two final states is used as the basis for the classification decision. Evaluate this model against the SST dev set. You can set up the model however you wish for this.

To submit:

  • Your call to TfBidirectionalRNNClassifier (so that we can see the hyperparmeters you chose).
  • Your average F1 score according to a classification_report on the dev set.

As above, we will not evaluate you based on how good your F1 score is. You just need to submit one. There is even value in seeing what really doesn't work, so low scores have interest!