Exploring the IGM Experts Forum Data: Confidence and Consensus

This is a follow up exploring the data that I collected in an earlier post. All of the data is from the IGM Experts Forum, which surveys a group of 51 leading economists on a variety of policy questions. A csv of all the data is available here, as are separate datasets of the questions and responses.

I'm especially interested in how confidence changes with the scale of a claim, so I use a few different techniques to look at that relationship. First, I look at confidence by vote type and find that economists seem to be more confident when they 'strongly agree' or 'strongly disagree'. Second, I find that confidence actually increases the further a view is from the median, although this is relationship is mainly driven by 25 votes out of a 7024 vote sample.

An earlier paper [1, PDF] on a portion of this data by Gordon and Dahl found that males and economists that were educated at the University of Chicago and MIT seemed to be more confident. I find less evidence of this in the newer data, although I lack the knowledge of statistics to say whether any of these differences are significant.

The main takeaway from this analysis is the amazing amount of consensus among leading economists. The mean and median distance away from the consensus responses are 0.63 and 0.45 points on a five point scale. Roughly 90 percent of the responses are within 1.5 units of the consensus for all questions. These results are consistend with Gordon and Dahl's earlier findings [2, URL].

Descriptive Statistics

First, let's look at some descriptive statistics for the responses:

In [430]:
%matplotlib inline

import pandas as pd
import matplotlib.pyplot as plt
import matplotlib

#pd.set_option('max_colwidth', 30)
pd.set_option('max_colwidth', 400)

matplotlib.rcParams['figure.figsize'] = (10.0, 8.0)
df_responses = pd.read_csv('output_all.csv')

cols = ['name', 'institution', 'qtitle','subquestion',
		 'qtext','vote', 'comments', 'median_vote']

# Summary of the string columns
df_responses.describe(include = ['O'])[cols]
name institution qtitle subquestion qtext vote comments median_vote
count 8402 8402 8402 8402 8402 8402 2921 8402
unique 51 7 134 3 199 9 2816 5
top José Scheinkman Chicago European Debt Question A Federal mandates that government purchases should be “buy American” unless there are exceptional circumstances, such as in the American Recovery and Reinvestment Act of 2009, have a significant positive impact on U.S. manufacturing employment. Agree -see background information here Agree
freq 199 1466 162 5686 51 2902 92 3877

Confidence Grouped by Vote Type

Next, I use the Pandas dataframe grouping function to look at confidence by vote type. Note that the Disagree and Strongly Disagree votes are much less common. Agree is the most common vote, followed by Uncertain and Strongly Agree. Uncertain has the lowest mean confidence at 4.3, possibly because it's weird to say that you're 'confidently uncertain'.

Overall, Strongly Agree and Strongly Disagree have higher mean and median confidences. One possible explanation is that economists are unwilling to step into the Strongly categories unless they feel that they have very good evidence. Another possibility is that this is an issue with the survey -- it's weird to say that you 'unconfidently strongly agree'.

In [431]:
# Initial grouping, just by vote.  

r_list = ['Strongly Disagree', 'Disagree', 'Uncertain', 'Agree', 'Strongly Agree']

filtered_vote = df_responses[df_responses['vote'].isin(r_list)]

filtered_vote.boxplot(column='confidence', by='vote', whis=[5.0,95.0])

                                 {'mean': 'mean', 
                                  'std': 'std', 
                                  'count': 'count',
                                  'median': 'median'}})
std count median mean
Agree 2.044886 2902 6 5.560992
Disagree 2.006320 992 6 5.530242
Strongly Agree 1.764728 1347 8 8.152190
Strongly Disagree 1.886012 314 8 7.843949
Uncertain 2.405343 1469 5 4.304969

Vote Distance from the Median

The above results are interesting, but what I'm more interested in is confidence as a claim becomes more controversial. Below, I construct a measure of vote distance from the median vote, then look at confidence grouped by distance from that median. I use the Pandas apply function below to assign a value ranging from 0 (Strongly Disagree) to 4 (Strongly Agree) to both the vote and median_vote columns. I then take the absolute value of the difference between each vote and the median_vote for each question to calculate the distance.

Confidence does increase the further the vote is from the median view, but relationship this is driven by 385 votes two points away, and 25 votes three points away out of a 7024 vote sample. It's possible that these confident yet controversial votes are from subject matter experts and have more information about a topic than the rest.

In [432]:
# Additional analyses, applying indicator column to quantify votes
# and sum/mean/count the confidence, grouped by vote_distance

def indicator(x):
	if x in r_list:
		return r_list.index(x)
		return None

df_responses['vote_num'] = df_responses['vote'].apply(indicator)
df_responses['median_num'] = df_responses['median_vote'].apply(indicator)
df_responses['vote_distance'] = abs(df_responses['median_num'] - df_responses['vote_num'])

grouped = df_responses.groupby('vote_distance').agg({'confidence':{'mean': 'mean', 
                                                                  'std': 'std', 

#Specifically note that there are only 25 votes with vote istance > 3, so this could be driven
#by a few experts that know something others don't  

df_responses.boxplot(column='confidence', by='vote_distance', whis=[5.0,95.0]) #bootstrap=1000

# Mean and Standard deviation

temp = df_responses.groupby('vote_distance').agg({'confidence': {'mean':'mean'}})

temp.plot.bar(yerr=grouped.loc[:, ('confidence', 'std')],  color='#7eb2fc')

std count mean
0 2.375436 3539 5.663747
1 2.507784 3075 6.103089
2 2.429312 385 6.202597
3 2.406934 25 7.720000

Making a Continuous Vote Column

To add some granularity to the above data, I combine the vote number column and the confidence column into one incremental column called incr_votenum. So a vote of Agree (vote_num = 3) at a confidence of 5 leads to a incr_votenum of 3.454 (3 + 5/11). The assumption I am making here is that confidence is a continuous measure between two votes, with an Agree vote of confidence 10 measuring less than a Strongly Agree vote at confidence 0. I'm not sure if this is a safe assumption to make, but I'm going to run with it.

I then calculate the median incr_votenum for each question, and the distance away from the median for each vote. A few example results are shown in the table below.

In [434]:
# Construct a continuous column, incorporating confidence into vote_num
# Divide by 11 so 10 confidence of agree > 0 confidence of strongly agree
df_responses['incr_votenum'] = df_responses['vote_num'] + df_responses['confidence'] / 11.0

# Median incr_votenum for each question:
df_responses['median_incrvotenum'] = df_responses.groupby(

# Calculate distance from median for each econ vote, less biased by outliers.  
df_responses['distance_median'] = abs(df_responses['median_incrvotenum'] - \

df_responses[df_responses['qtitle'] == 'Brexit II'][
     'incr_votenum', 'median_incrvotenum', 'distance_median']].head()
qtitle subquestion vote_num confidence incr_votenum median_incrvotenum distance_median
0 Brexit II Question A 2 4 2.363636 3.363636 1.000000
1 Brexit II Question B 3 5 3.454545 3.272727 0.181818
198 Brexit II Question A 3 5 3.454545 3.363636 0.090909
199 Brexit II Question B 3 5 3.454545 3.272727 0.181818
397 Brexit II Question A 3 3 3.272727 3.363636 0.090909

Visualizing the Spread of the Votes

The following boxplot shows the distance from the median for all votes, using the new incremental vote measure. It's pretty amazing that the mean and median distance away from the consensus are only 0.454 and 0.628 respectively. That's an impressive amount of consensus. The whiskers on the boxplot cover the 90 percent of the data that fall within roughly 1.6 points of the consensus vote on a scale from 0 to 5.

The histogram also shows a suprising amount of consensus, although it also shows a second peak around a difference of 1.0, which is the difference between two bordering answers, e.g. the distance from Uncertain to Agree.

In [457]:
# Boxplot, showing all vote distances from median  
df_responses.boxplot(column='distance_median', whis=[5.0,95.0], return_type='dict')

df_responses.hist(column='distance_median', bins=40)

print 'Median: ' + str(df_responses['distance_median'].median())
print 'Mean: ' + str(df_responses['distance_median'].mean())
print 'Stdev: ' + str(df_responses['distance_median'].std())
Median: 0.454545454545
Mean: 0.628856906192
Stdev: 0.556692286667

Which answers are furthest from the median response?

In [451]:
# Answers that are furthest from median:
df_responses[df_responses['distance_median'] >= 2.75][['name', 'qtitle','subquestion', 'qtext', 'vote',
                        'confidence', 'median_vote', 'distance_median']].sort_values(
name qtitle subquestion qtext vote confidence median_vote distance_median
6497 Alberto Alesina Bureau of Labor Statistics Question A By providing important measures of US economic performance — including employment, consumer prices, wages, job openings, time allocation in households, and productivity — the Bureau of Labor Statistics creates social benefits that exceed its annual cost of roughly $610 million. Disagree 2 Strongly Agree 3.454545
1914 Angus Deaton Oil Price Speculation Question A Large movements in monthly oil prices, either up or down, are driven primarily by speculators, as opposed to changes in the current (and planned) supply or demand for oil. Strongly Agree 10 Disagree 3.363636
6498 Alberto Alesina Bureau of Labor Statistics Question B Cuts in BLS spending would likely involve net social costs because potential declines in the quality of data, and thus their usefulness to researchers and decision makers, would exceed any budget savings. Disagree 2 Strongly Agree 3.318182
2062 Angus Deaton Price Gouging Question A Connecticut should pass its Senate Bill 60,which statesthat during a “severe weather event emergency, no person within the chain of distribution of consumer goods and services shall sell or offer to sell consumer goods or services for a price that is unconscionably excessive.” Strongly Agree 9 Disagree 3.272727
4792 Caroline Hoxby Economic Stimulus Question A Because of the American Recovery and Reinvestment Act of 2009, the U.S. unemployment rate was lower at the end of 2010 than it would have been without the stimulus bill. Strongly Disagree 5 Agree 3.227273
7478 Liran Einav Diversification Question A In general, absent any inside information, an equity investor can expect to do better by choosing a well-diversified, low-cost index fund than by picking a few stocks. Disagree 7 Strongly Agree 3.090909
8371 Hilary Hoynes Bah, Humbug Question A Giving specific presents as holiday gifts is inefficient, because recipients could satisfy their preferences much better with cash. Strongly Agree 10 Disagree 3.090909
5287 Edward Lazear Carbon Tax Question A A tax on the carbon content of fuels would be a less expensive way to reduce carbon-dioxide emissions than would a collection of policies such as “corporate average fuel economy” requirements for automobiles. Disagree 5 Strongly Agree 3.045455
2064 Angus Deaton Ticket Resale Question A Laws that limit the resale of tickets for entertainment and sports events make potential audience members for those events worse off on average. Strongly Disagree 7 Agree 2.909091
6665 Alberto Alesina Economic Stimulus Question B Taking into account all of the ARRA’s economic consequences — including the economic costs of raising taxes to pay for the spending, its effects on future spending, and any other likely future effects — the benefits of the stimulus will end up exceeding its costs. Strongly Disagree 4 Uncertain 2.863636
1982 Angus Deaton Surge Pricing Question A Using surge pricing to allocate transportation services — such asUberdoes with its cars — raises consumer welfare through various potential channels, such as increasing the supply of those services, allocating them to people who desire them the most, and reducing search and queuing costs. Strongly Disagree 10 Agree 2.818182
6570 Alberto Alesina Surge Pricing Question A Using surge pricing to allocate transportation services — such asUberdoes with its cars — raises consumer welfare through various potential channels, such as increasing the supply of those services, allocating them to people who desire them the most, and reducing search and queuing costs. Strongly Disagree 10 Agree 2.818182
4793 Caroline Hoxby Economic Stimulus Question B Taking into account all of the ARRA’s economic consequences — including the economic costs of raising taxes to pay for the spending, its effects on future spending, and any other likely future effects — the benefits of the stimulus will end up exceeding its costs. Strongly Disagree 5 Uncertain 2.772727

Which questions are most controversial?

As a measure of how controversial a question is, I take the standard deviation of the incremental vote number (incr_votenum). I include both a table of the questions, and a boxplot below.

In [442]:
# Which questions are most controversial?

# Calculating standard deviation, grouped by each question:
grouped_incrvotenum = df_responses.groupby(['qtitle', 'subquestion','qtext'], as_index = False) \
                                        .agg({'incr_votenum': {'std': 'std'}})

# Visualize the spread of responses using a boxplot
qs = grouped_incrvotenum[grouped_incrvotenum.loc[:, ('incr_votenum', 'std')] > 1.05][['qtitle','subquestion']]
qs_most = pd.merge(qs, df_responses, on=['qtitle', 'subquestion'], how='inner')
qs_most.boxplot(column='incr_votenum',by=['qtitle','subquestion'], whis=[5.0,95.0], rot=90)  

# Show table of questions with a stdev of incr_votenum > 1.05
grouped_incrvotenum[grouped_incrvotenum.loc[:, ('incr_votenum', 'std')] > 1.05].sort_values(
qtitle subquestion qtext incr_votenum
134 Poverty and Measurement Question A The association between health and economic growth in poor countries primarily involves faster growth generating better health, rather than the other way around. 1.149829
128 Oil Price Speculation Question A Large movements in monthly oil prices, either up or down, are driven primarily by speculators, as opposed to changes in the current (and planned) supply or demand for oil. 1.135301
135 Poverty and Measurement Question B The decline in the fraction of people with incomes under, say, $1 per day is a good measure of whether well-being is improving among low-income populations. 1.123439
162 Student Credit Risk Question A Conventional economic reasoning suggests that it would be a good policy to enact therecent Senate billthat would let undergraduate students borrow through the government Stafford program at interest rates equivalent to the primary credit rates offered to banks through the Federal Reserve's discount window. 1.117773
132 Patents Question B Within the software industry, the US patent system makes consumers better off than they would be in the absence of patents. 1.114383
50 Economic Stimulus Question B Taking into account all of the ARRA’s economic consequences — including the economic costs of raising taxes to pay for the spending, its effects on future spending, and any other likely future effects — the benefits of the stimulus will end up exceeding its costs. 1.111368
165 Supplying Kidneys Question A A market that allows payment for human kidneys should be established on a trial basis to help extend the lives of patients with kidney disease. 1.073536
33 Christmas Spending Question A An annual December spending surge on parties, gift-giving and personal travel delivers net social benefits. 1.064723
76 Fracking (revisited) Question A New technology for fracking natural gas, by lowering energy costs in the United States, will make US industrial firms more cost competitive and thus significantly stimulate the growth of US merchandise exports. (The experts panel previously voted on this question on May 23, 2012. Those earlier results can be foundhere.) 1.061360
48 Early Education Question A Using government funds to guarantee preschool education for four-year olds would yield a much lower social return than the ones achieved by the most highly touted targeted preschool initiatives. 1.060312
75 Fracking Question A New technology for fracking natural gas, by lowering energy costs in the United States, will make US industrial firms more cost competitive and thus significantly stimulate the growth of US merchandise exports. 1.056384
175 Ten-year Budgets Question A Because federalspending on Medicare and Medicaidwill continue to grow under current policy beyond the 10-year window of most political budget debates, it is easy for a politician to devise a budget plan that would reduce federal deficits over the next decade without really making the U.S. fiscally sustainable. 1.056000

Which questions are least controversial?

In [444]:
# Which questions are least controversial?

# Select all data for questions with qtitle and subquestion by merging 
qs_least = grouped_incrvotenum[grouped_incrvotenum.loc[:, ('incr_votenum', 'std')] < 0.6][['qtitle','subquestion']]
qs_least_df = pd.merge(qs_least, df_responses, on=['qtitle', 'subquestion'], how='inner')

# Visualize boxplot and table
qs_least_df.boxplot(column='incr_votenum',by=['qtitle','subquestion'], rot=90, whis=[5.0,95.0]) 
grouped_incrvotenum[grouped_incrvotenum.loc[:, ('incr_votenum', 'std')] < 0.6].sort_values(
qtitle subquestion qtext incr_votenum
197 Vaccines Question A Declining to be vaccinated against contagious diseases such as measles imposes costs on other people, which is a negative externality. 0.356966
84 Gold Standard Question A If the US replaced its discretionary monetary policy regime with a gold standard, defining a "dollar" as a specific number of ounces of gold, the price-stability and employment outcomes would be better for the average American. 0.417698
91 Healthcare Question A There are no consequential distortions created by the tax preference that favors obtaining health insurance through employers. 0.517269
85 Gold Standard Question B There are many factors besides US inflation risk that influence the current dollar price of gold. 0.525888
146 Rent Control Question A Local ordinances that limit rent increases for some rental housing units, such as inNew YorkandSan Francisco, have had a positive impact over the past three decades on the amount and quality of broadly affordable rental housing in cities that have used them. 0.551146
37 Congress and Monetary Policy Question A Legislationintroducedin Congress would require the Federal Reserve to "submit to the appropriate congressional committees…a Directive Policy Rule", which shall "describe the strategy or rule of the Federal Open Market Committee for the systematic quantitative adjustment of the Policy Instrument Target to respond to a change in the Intermediate Policy Inputs." Should the Fed deviate from the ru... 0.554602
108 Laffer Curve Question B A cut in federal income tax rates in the US right now would raise taxable income enough so that the annual total tax revenue would be higher within five years than without the tax cut. 0.556231
45 Dynamic Scoring Question A Changing federal income tax rates, or the income bases to which those rates apply, can affect federal tax revenues partly by altering people’s behavior, and thus their actual or reported incomes. 0.560967
120 Monetary Policy Question A All else equal, the Fed's new plan to increase the maturity of its Treasury holdings will boost expected real GDP growth for calendar year 2012 by at least one percentage point. 0.573926
174 Taxing Capital and Labor Question C Although they do not always agree about the precise likely effects of different tax policies, another reason why economists often give disparate advice on tax policy is because they hold differing views about choices between raising average prosperity and redistributing income. 0.581531
171 Taxi Competition Question A Letting car services such as Uber or Lyft compete with taxi firms on equal footing regarding genuine safety and insurance requirements, but without restrictions on prices or routes, raises consumer welfare. 0.584899
124 Nash Equilibrium Question A Behavior in many complex and seemingly intractable strategic settings can be understood more clearly by working out what each party in the game will choose to do if they realize that the other parties will be solving the same problem. This insight has helped us understand behavior as diverse as military conflicts, price setting by competing firms and penalty kicking in soccer. 0.585807
196 Universal Basic Income Question A Granting every American citizen over 21-years old auniversal basic income of $13,000 a year— financed by eliminating all transfer programs (including Social Security, Medicare, Medicaid, housing subsidies, household welfare payments, and farm and corporate subsidies) — would be a better policy than the status quo. 0.586828

Which economists give more controversial responses?

In [445]:
# Group by economist, calcluate mean distance from median
grouped_econstd = df_responses.groupby(['name', 'institution']).agg({'distance_median': {'mean': 'mean'}})

grouped_econstd[grouped_econstd.loc[:, ('distance_median', 'mean')] > 0.75].sort_values(
name institution
Alberto Alesina Harvard 0.904429
Angus Deaton Princeton 0.891251
Caroline Hoxby Stanford 0.889181
Austan Goolsbee Chicago 0.791797
Luigi Zingales Chicago 0.770085

Which economists give the least controversial responses?

In [446]:
# Which economists give the least controversial responses? 

grouped_econstd[grouped_econstd.loc[:, ('distance_median', 'mean')] < 0.50].sort_values(
name institution
James Stock Harvard 0.431818
Amy Finkelstein MIT 0.462121
Eric Maskin Harvard 0.471361
Raj Chetty Harvard 0.474530

Do any institutions give controversial responses?

Here's what Gordon and Dahl had to say about differences between institutions using the 2012 question sample [1, PDF]:

Respondents are dramatically more confident when the academic literature on the topic is large. Not surprisingly, experts on a subject are much more confident about their answers. The middle-aged cohort (the one closest to the current literature) is the most confident, while the oldest (and wisest) cohort is the least confident. Men and those who have worked in Washington do show some tendency to be more confident. Respondents who got their degrees at Chicago are far more confident than the other respondents, with almost as strong an effect for respondents with PhDs from MIT and to a lesser extent from Harvard. Respondents now employed at Yale and to a lesser degree Princeton, MIT, and Stanford seem to be more confident.

It doesn't seem like any institution sticks out based on this newer data, but maybe with more advanced statistical techniques it might be possible to find something significant.

In [447]:
# Group by institution, calculate mean and stdev of the distance from median response
grouped_inststd = df_responses.groupby(['institution']).agg({'distance_median':
                                                             {'mean': 'mean', 

df_responses.boxplot(column='distance_median', by='institution', whis=[5.0,95.0])

std mean
Stanford 0.572456 0.673072
Chicago 0.561737 0.666043
Princeton 0.586784 0.662991
MIT 0.554587 0.621699
Harvard 0.558052 0.596441
Yale 0.537219 0.593209
Berkeley 0.526592 0.584685

Are any institutions more confident than others?

Again, although there are differences in the mean distance from the consensus view, all the standard deviations overlap, so I don't think there is anything significant here. Note that Gordon and Dahl also looked at where economists where educated, rather than just where they were employed, and found differences in confidence based on that metric.

In [448]:
#Are any institutions more confident than others?
df_responses.boxplot(column='confidence', by='institution', whis=[5.0,95.0]) 

grouped_conf = df_responses.groupby(['institution']).agg(
                     {'mean': 'mean',

std median mean
Princeton 2.168625 7 6.363905
Berkeley 2.360717 6 6.019286
Yale 2.457948 6 6.009033
MIT 2.094831 6 5.909300
Stanford 2.555858 6 5.859665
Harvard 2.330881 6 5.725877
Chicago 2.789926 6 5.580745

Are male economists more confident?

Gordon and Dahl (2012) noted more confidence among male economists:

The only statistically significant deviation from homogeneous views, therefore, is less caution among men in expressing an opinion, perhaps due to a greater “expert bias.” Personality differences rather than different readings of the existing evidence would then explain these gender effects.

This relationship seems to be less obvious with this expanded dataset. I'm not re-creating their analysis, though, so the difference might still be there if I were to use the controls that they do.

In [449]:
women = ['Amy Finkelstein', 'Hilary Hoynes', 'Pinelopi Goldberg', 
         'Judith Chevalier', 'Caroline Hoxby', 'Nancy Stokey', 
        'Marianne Bertrand', 'Cecilia Rouse', 'Janet Currie', 
         'Claudia Goldin', 'Katherine Baicker']

# Set true/false column based on sex
df_responses['female'] = df_responses['name'].isin(women)

# Boxplot grouped by sex 
df_responses.boxplot(column='confidence', by='female', whis=[5.0,95.0])

# Table, stats grouped by sex
df_responses.groupby(['female']).agg({'confidence': {'mean': 'mean', 'std':'std', 'median':'median'}})
std median mean
False 2.398254 6 5.929568
True 2.661413 6 5.729814