__author__ = "Christopher Potts"
__version__ = "CS224u, Stanford, Spring 2018 term"
This homework covers material from the sentiment classification unit. The primary value of doing the work is that it provides more hands-on experience with the dataset and the models we explored. All the code you write has potential value in the bake-off for this unit as well.
Submission URL: https://canvas.stanford.edu/courses/83399/quizzes/50657
Socher et al. 2013 compare against a Naive Bayes baseline with bigram features. See how close you can come to reproducing the performance of this model on the binary, root-only problem (values in the rightmost column of their Table 1, rows 1 and 3).
Specific tasks:
Write a bigrams feature function, on the model of unigrams_phi
. Call this bigrams_phi
. In writing this function, ensure that each example is padded with start and end symbols (say, <S>
and </S>
), so that these contexts are properly reflected in the feature space.
Write a function fit_nb_classifier_with_crossvalidation
that serves as a wrapper for sklearn.naive_bayes.MultinomialNB
and searches over these values for the smoothing parameter alpha
: [0.1, 0.5, 1.0, 2.0]
, using 3-fold cross-validation.
Use sst.experiment
to run the experiments, assessing against dev_reader
.
To submit:
bigrams_phi
fit_nb_classifier
sst.experiment
sst.experiment
reported.A note on performance: in our experience, the bigrams Naive Bayes model gets around 0.75. It's fine to submit answers with comparable numbers; the Socher et al. baselines are very strong. We're not evaluating how good your model is; we want to see your code, and we're interested to see what the range of F1 scores is across the whole class.
In the section Distributed representations as features, we looked at a baseline for the binary SST problem in which each example is modeled as the sum of its 50-dimensional GloVe representations. A LogisticRegression
model was used for prediction. A neural network might do better here, since there might be complex relationships between the input feature dimensions that a linear classifier can't learn.
To address this question, rerun the experiment with tf_shallow_neural_classifier.TfShallowNeuralClassifier
as the classifier. Specs:
sst.experiment
to conduct the experiment.tf.nn.tanh
or tf.nn.relu
.TfShallowNeuralClassifier
, use the defaults.)To submit:
sst.experiment
.sst._experiment
prints.)No need to include your supporting code.
We're not evaluating the quality of your model. (We've specified the protocols completely, but there will still be a lot of variation in the results.) However, the primary goal of this question is to get you thinking more about this strikingly good baseline feature representation scheme for SST, so we're sort of hoping you feel compelled to try out variations on your own.
The auxiliary notebook tensorflow_models.ipynb
subclasses TfRNNClassifier with a bidirectional RNN. In this model, the RNN is run in both directions and the concatenation of the two final states is used as the basis for the classification decision. Evaluate this model against the SST dev set. You can set up the model however you wish for this.
To submit:
TfBidirectionalRNNClassifier
(so that we can see the hyperparmeters you chose).classification_report
on the dev
set.As above, we will not evaluate you based on how good your F1 score is. You just need to submit one. There is even value in seeing what really doesn't work, so low scores have interest!