Reviewer Expertise

23rd July 2014 Neil D. Lawrence

In response to a question from Christoph Lippert about the expertise of reviewers, this notebook loads in reviewer expertise and summarizes it in a couple of different ways.

In [23]:
import cmtutils
import pylab as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = (10., 8.)
plt.rcParams['font.size'] = 16

Load in the reviewer data from 24th July alongside the subject areas.

In [1]:
reviewers = cmtutils.reviewers(filename='2014-07-24_reviewer_export.xls',
                               subject_file='2014-07-24_reviewerSubjectAreas.xls')
Loaded Users.
Loaded Reviewer Subjects.

A little helper function for histogramming papers.

In [106]:
def histogram_papers(papers, title):
    papers = papers.fillna(0)
    a= plt.hist(papers, bins=papers.max()-papers.min(), range=(papers.min(), papers.max()), align='mid')
    plt.title(title)
    plt.grid(False)
    print papers.max()

Reviewers: Papers Since 2007

Now let's start plotting. First we histogram the reviewing body showing the number of reviewers who've had the corresponding numbers of papers (since 2007).

In [109]:
histogram_papers(reviewers.users[reviewers.users.IsMetaReviewer=='No'].PapersSince2007,
                 'Reviewers: Number of NIPS papers since 2007')
22.0

Reviewers: Papers Since 2012

Same idea, but more recent expertise, how many papers have reviewers had since 2012?

In [110]:
histogram_papers(reviewers.users[reviewers.users.IsMetaReviewer=='No'].PapersSince2012,
                 'Reviewers: Number of NIPS papers since 2012')
10.0

Area Chairs: Papers Since 2007

Now the same for Area Chairs, how many papers have they had since 2007?

In [103]:
histogram_papers(reviewers.users[reviewers.users.IsMetaReviewer=='Yes'].PapersSince2007,
                 'Area Chairs: Number of NIPS papers since 2007')

Area Chairs: Papers Since 2012

And finally the recent experience of Area Chairs. How many papers have they had since 2012?

In [104]:
histogram_papers(reviewers.users[reviewers.users.IsMetaReviewer=='Yes'].PapersSince2012,
                 'Area Chairs: Number of NIPS papers since 2012')