My Work

Extracting the collective wisdom in probabilistic judgments (pdf)

(Under Review, Supplemental Material, Presentation, Presentation (short))

Experts often disagree in their judgments on the likelihood of an event. How should we combine individual estimates for maximum accuracy? I develop a novel algorithm for judgment aggregation. Experts are asked to report a prediction and an estimate on the average of others' predictions. The algorithm uses the latter to produce a consistent estimator for the unknown probability. Evidence suggests that the algorithm performs well especially in difficult aggregation problems where individuals disagree greatly.

Incentives for self-extremized expert judgments to alleviate the shared-information problem (pdf)

(Revised & Resubmitted at Decision's Special Issue on the Wisdom of Crowds, R scripts)

Simple average of forecasts is a common method to aggregate expert judgments. However, experts' shared information is over-represented in the average forecast. I propose a simple incentive-based solution: Experts are grouped with non-experts and rewarded for group accuracy. As a result, experts extremize their forecasts towards away from the shared information to boost crowd accuracy.

Peer prediction markets to elicit unverifiable information (pdf)

(joint with A. Baillon and S. van der Zee)

We propose a prediction market to incentivize effort in reporting unverifiable judgments, such as personal experiences or long-term forecasts. Incentives are aligned such that Bayesian agents exert mental effort and trade accordingly, indirectly revealing their true judgment in the process. Evidence from two experimental studies indicate that peer predicition markets elicit high-quality subjective data.

Robust recalibration of aggregate probability forecasts using meta-beliefs

(in progress, joint with T. Wilkening)

Average probabilistic forecast of a crowd is typically too conservative. However, blind extremization may introduce further miscalibration in some tasks. We develop a recalibration method, which uses forecasters' meta-beliefs to improve accuracy.