Homework 3: Relation extraction using distant supervision

In [1]:
__author__ = "Bill MacCartney"
__version__ = "CS224U, Stanford, Spring 2019"

Overview

This homework and associated bake-off are devoted to the developing really effective relation extraction systems using distant supervision.

As with the previous assignments, this notebook first establishes a baseline system. The initial homework questions ask you to create additional baselines and suggest areas for innovation, and the final homework question asks you to develop an original system for you to enter into the bake-off.

Set-up

See the first notebook in this unit for set-up instructions.

In [2]:
import os
import rel_ext
from sklearn.linear_model import LogisticRegression

As usual, we unite our corpus and KB into a dataset, and create some splits for experimentation:

In [3]:
rel_ext_data_home = os.path.join('data', 'rel_ext_data')
In [4]:
corpus = rel_ext.Corpus(os.path.join(rel_ext_data_home, 'corpus.tsv.gz'))
In [5]:
kb = rel_ext.KB(os.path.join(rel_ext_data_home, 'kb.tsv.gz'))
In [6]:
dataset = rel_ext.Dataset(corpus, kb)

You are not wedded to this set-up for splits. The bake-off will be conducted on a previously unseen test-set, so all of the data in dataset is fair game:

In [7]:
splits = dataset.build_splits(
    split_names=['tiny', 'train', 'dev'],
    split_fracs=[0.01, 0.79, 0.20],
    seed=1)
In [8]:
splits
Out[8]:
{'tiny': Corpus with 3,474 examples; KB with 445 triples,
 'train': Corpus with 263,285 examples; KB with 36,191 triples,
 'dev': Corpus with 64,937 examples; KB with 9,248 triples,
 'all': Corpus with 331,696 examples; KB with 45,884 triples}

Baseline

In [9]:
def simple_bag_of_words_featurizer(kbt, corpus, feature_counter):
    for ex in corpus.get_examples_for_entities(kbt.sbj, kbt.obj):
        for word in ex.middle.split(' '):
            feature_counter[word] += 1
    for ex in corpus.get_examples_for_entities(kbt.obj, kbt.sbj):
        for word in ex.middle.split(' '):
            feature_counter[word] += 1
    return feature_counter
In [10]:
featurizers = [simple_bag_of_words_featurizer]
In [11]:
model_factory = lambda: LogisticRegression(fit_intercept=True, solver='liblinear')
In [12]:
baseline_results = rel_ext.experiment(
    splits,
    train_split='train',
    test_split='dev',
    featurizers=featurizers,
    model_factory=model_factory,
    verbose=True)
relation              precision     recall    f-score    support       size
------------------    ---------  ---------  ---------  ---------  ---------
adjoins                   0.853      0.391      0.690        340       5716
author                    0.790      0.532      0.720        509       5885
capital                   0.562      0.189      0.404         95       5471
contains                  0.793      0.596      0.744       3904       9280
film_performance          0.781      0.563      0.725        766       6142
founders                  0.779      0.400      0.655        380       5756
genre                     0.686      0.141      0.387        170       5546
has_sibling               0.878      0.230      0.562        499       5875
has_spouse                0.853      0.323      0.643        594       5970
is_a                      0.689      0.249      0.509        497       5873
nationality               0.571      0.186      0.404        301       5677
parents                   0.871      0.542      0.777        312       5688
place_of_birth            0.686      0.206      0.468        233       5609
place_of_death            0.459      0.107      0.277        159       5535
profession                0.570      0.215      0.428        247       5623
worked_at                 0.693      0.252      0.513        242       5618
------------------    ---------  ---------  ---------  ---------  ---------
macro-average             0.720      0.320      0.557       9248      95264

Studying model weights might yield insights:

In [13]:
rel_ext.examine_model_weights(baseline_results)
Highest and lowest feature weights for relation adjoins:

     2.552 Córdoba
     2.465 Taluks
     2.426 Valais
     ..... .....
    -1.175 based
    -1.287 other
    -1.431 America

Highest and lowest feature weights for relation author:

     2.791 author
     2.397 wrote
     2.270 by
     ..... .....
    -1.978 or
    -2.115 directed
    -7.687 dystopian

Highest and lowest feature weights for relation capital:

     3.250 capital
     1.713 city
     1.552 posted
     ..... .....
    -1.343 and
    -2.718 Province
    -2.725 Isfahan

Highest and lowest feature weights for relation contains:

     2.547 southwestern
     2.040 borders
     2.007 affiliated
     ..... .....
    -2.419 2002
    -2.778 Isfahan
    -2.875 band

Highest and lowest feature weights for relation film_performance:

     4.198 starring
     3.842 co-starring
     3.283 movie
     ..... .....
    -2.220 Anjaani
    -2.220 Anjaana
    -3.996 double

Highest and lowest feature weights for relation founders:

     3.874 founded
     3.645 founder
     3.203 co-founder
     ..... .....
    -1.411 series
    -1.477 state
    -1.892 band

Highest and lowest feature weights for relation genre:

     2.804 series
     2.574 album
     2.405 movie
     ..... .....
    -1.482 ;
    -1.766 at
    -2.142 follows

Highest and lowest feature weights for relation has_sibling:

     5.312 brother
     4.030 sister
     3.043 nephew
     ..... .....
    -1.285 engineer
    -1.290 from
    -1.357 Jacob

Highest and lowest feature weights for relation has_spouse:

     5.395 wife
     4.599 husband
     4.389 widow
     ..... .....
    -1.350 on
    -1.625 engineer
    -1.646 Terri

Highest and lowest feature weights for relation is_a:

     2.940 family
     2.882 genus
     2.585 
     ..... .....
    -1.647 kamut
    -1.679 on
    -2.949 hibiscus

Highest and lowest feature weights for relation nationality:

     2.661 born
     1.974 president
     1.966 caliph
     ..... .....
    -1.344 state
    -1.395 and
    -1.677 American

Highest and lowest feature weights for relation parents:

     5.282 son
     4.750 daughter
     4.418 father
     ..... .....
    -1.737 Jacob
    -1.980 Jahangir
    -2.552 Kelly

Highest and lowest feature weights for relation place_of_birth:

     3.729 born
     3.125 birthplace
     2.820 mayor
     ..... .....
    -1.390 or
    -1.523 and
    -2.121 Oldham

Highest and lowest feature weights for relation place_of_death:

     2.826 died
     1.934 under
     1.870 where
     ..... .....
    -1.294 that
    -1.301 and
    -1.444 Siege

Highest and lowest feature weights for relation profession:

     3.103 
     2.394 American
     2.391 philosopher
     ..... .....
    -1.417 York
    -1.713 elder
    -2.205 on

Highest and lowest feature weights for relation worked_at:

     3.318 professor
     3.124 president
     2.824 CEO
     ..... .....
    -1.201 then-associate
    -1.254 NASA
    -1.650 or

Homework questions

Please embed your homework responses in this notebook, and do not delete any cells from the notebook. (You are free to add as many cells as you like as part of your responses.)

Different model factory [1 point]

The code in rel_ext makes it very easy to experiment with other classifier models: one need only redefine the model_factory argument. This question asks you to assess a Support Vector Classifier.

To submit: A call to rel_ext.experiment training on the 'train' part of splits and assessing on its dev part, with featurizers as defined above in this notebook and the model_factory set to one based in an SVC with kernel='linear' and all other arguments left with default values.

In [ ]:
 

Directional unigram features [2 points]

The current bag-of-words representation makes no distinction between "forward" and "reverse" examples. But, intuitively, there is big difference between X and his son Y and Y and his son X. This question asks you to modify simple_bag_of_words_featurizer to capture these differences.

To submit:

  1. A feature function directional_bag_of_words_featurizer that is just like simple_bag_of_words_featurizer except that it distinguishes "forward" and "reverse". To do this, you just need to mark each word feature for whether it is derived from a subject–object example or from an object–subject example. The precise nature of the mark you add for the two cases doesn't make a difference to the model.

  2. The macro-average F-score on the dev set that you obtain from running rel_ext.experiment with directional_bag_of_words_featurizer as the only featurizer. (Aside from this, use all the default values for experiment as exemplified above in this notebook.)

  3. rel_ext.experiment returns some of the core objects used in the experiment. How many feature names does the vectorizer have for the experiment run in the previous step? (Note: we're partly asking you to figure out how to get this value by using the sklearn documentation, so please don't ask how to do it on Piazza!)

In [ ]:
 

The part-of-speech tags of the "middle" words [2 points]

Our corpus distribution contains part-of-speech (POS) tagged versions of the core text spans. Let's begin to explore whether there is information in these sequences, focusing on middle_POS.

To submit:

  1. A feature function middle_bigram_pos_tag_featurizer that is just like simple_bag_of_words_featurizer except that it creates a feature for bigram POS sequences. For example, given

    The/DT dog/N napped/V

    we obtain the list of bigram POS sequences

    b = ['<s> DT', 'DT N', 'N V', 'V </s>'].

    Of course, middle_bigram_pos_tag_featurizer should return count dictionaries defined in terms of such bigram POS lists, on the model of simple_bag_of_words_featurizer.

    Don't forget the start and end tags, to model those environments properly!

  2. The macro-average F-score on the dev set that you obtain from running rel_ext.experiment with middle_bigram_pos_tag_featurizer as the only featurizer. (Aside from this, use all the default values for experiment as exemplified above in this notebook.)

Note: To parse middle_POS, one splits on whitespace to get the word/TAG pairs. Each of these pairs s can be parsed with s.rsplit('/', 1).

In [ ]:
 

Your original system [4 points]

There are many options, and this could easily grow into a project. Here are a few ideas:

  • Try out different classifier models, from sklearn and elsewhere.
  • Add a feature that indicates the length of the middle.
  • Augment the bag-of-words representation to include bigrams or trigrams (not just unigrams).
  • Introduce features based on the entity mentions themselves.
  • Experiment with features based on the context outside (rather than between) the two entity mentions — that is, the words before the first mention, or after the second.
  • Try adding features which capture syntactic information, such as the dependency-path features used by Mintz et al. 2009. The NLTK toolkit contains a variety of parsing algorithms that may help.
  • The bag-of-words representation does not permit generalization across word categories such as names of people, places, or companies. Can we do better using word embeddings such as GloVe?
  • Consider adding features based on WordNet synsets. Here's a little code to get you started with that:
    from nltk.corpus import wordnet as wn
    dog_compatible_synsets = wn.synsets('dog', pos='n')

Bake-off [1 point]

For the bake-off, we will release a test set right after class on April 29. The announcement will go out on Piazza. You will evaluate your custom model from the previous question on these new datasets using the function rel_ext.bake_off_experiment. Rules:

  1. Only one evaluation is permitted.
  2. No additional system tuning is permitted once the bake-off has started.

To enter the bake-off, upload this notebook on Canvas:

https://canvas.stanford.edu/courses/99711/assignments/187248

The cells below this one constitute your bake-off entry.

People who enter will receive the additional homework point, and people whose systems achieve the top score will receive an additional 0.5 points. We will test the top-performing systems ourselves, and only systems for which we can reproduce the reported results will win the extra 0.5 points.

The bake-off will close at 4:30 pm on May 1. Late entries will be accepted, but they cannot earn the extra 0.5 points. Similarly, you cannot win the bake-off unless your homework is submitted on time.

In [14]:
# Enter your bake-off assessment code in this cell. 
# Please do not remove this comment.
In [15]:
# On an otherwise blank line in this cell, please enter
# your macro-average f-score (an F_0.5 score) as reported 
# by the code above. Please enter only a number between 
# 0 and 1 inclusive. Please do not remove this comment.