__author__ = "Christopher Potts" __version__ = "CS224u, Stanford, Spring 2018 term"
This homework covers material from the sentiment classification unit. The primary value of doing the work is that it provides more hands-on experience with the dataset and the models we explored. All the code you write has potential value in the bake-off for this unit as well.
Submission URL: https://canvas.stanford.edu/courses/83399/quizzes/50657
Socher et al. 2013 compare against a Naive Bayes baseline with bigram features. See how close you can come to reproducing the performance of this model on the binary, root-only problem (values in the rightmost column of their Table 1, rows 1 and 3).
Write a bigrams feature function, on the model of
unigrams_phi. Call this
bigrams_phi. In writing this function, ensure that each example is padded with start and end symbols (say,
</S>), so that these contexts are properly reflected in the feature space.
Write a function
fit_nb_classifier_with_crossvalidation that serves as a wrapper for
sklearn.naive_bayes.MultinomialNB and searches over these values for the smoothing parameter
[0.1, 0.5, 1.0, 2.0], using 3-fold cross-validation.
sst.experiment to run the experiments, assessing against
A note on performance: in our experience, the bigrams Naive Bayes model gets around 0.75. It's fine to submit answers with comparable numbers; the Socher et al. baselines are very strong. We're not evaluating how good your model is; we want to see your code, and we're interested to see what the range of F1 scores is across the whole class.
In the section Distributed representations as features, we looked at a baseline for the binary SST problem in which each example is modeled as the sum of its 50-dimensional GloVe representations. A
LogisticRegression model was used for prediction. A neural network might do better here, since there might be complex relationships between the input feature dimensions that a linear classifier can't learn.
To address this question, rerun the experiment with
tf_shallow_neural_classifier.TfShallowNeuralClassifier as the classifier. Specs:
sst.experimentto conduct the experiment.
TfShallowNeuralClassifier, use the defaults.)
No need to include your supporting code.
We're not evaluating the quality of your model. (We've specified the protocols completely, but there will still be a lot of variation in the results.) However, the primary goal of this question is to get you thinking more about this strikingly good baseline feature representation scheme for SST, so we're sort of hoping you feel compelled to try out variations on your own.
The auxiliary notebook
tensorflow_models.ipynb subclasses TfRNNClassifier with a bidirectional RNN. In this model, the RNN is run in both directions and the concatenation of the two final states is used as the basis for the classification decision. Evaluate this model against the SST dev set. You can set up the model however you wish for this.
TfBidirectionalRNNClassifier(so that we can see the hyperparmeters you chose).
As above, we will not evaluate you based on how good your F1 score is. You just need to submit one. There is even value in seeing what really doesn't work, so low scores have interest!