Blinded Versus Unblinded Review: A Field Study Comparing the Equity of Peer-Review

Blinded review is often proposed as a solution to potential inequities in meritbased evaluations. Yet, an equitable merit-based evaluation should also be reliable and valid. We evaluated the equity of blinded and unblinded review in a high-stakes field experiment during peer-review of submissions for an academic conference. Each conference submission received double-blind peer-reviews with reviewer and author identities withheld and single-blind peer-reviews with only reviewer identities withheld. We found that both systems yielded evaluations with moderate reliability after averaging across three or more independent reviewers. This level of reliability resulted in the review systems agreeing on 40% of the top submissions. This agreement level suggests that some differences between review systems emerge from noisy human judgment. The comparison also revealed that single-blind reviews favored senior coauthors and disfavored Asian (versus White) authors more than double-blind review. Double-blind reviews slightly favored male authors more than single-blind reviews. For submissions selected for presentation, neither author characteristics nor the review process consistently predicted judged talk quality or popularity. Single- and double-blind review scores similarly predicted subsequent publication. These results imply that preferences based on authors’ characteristics during single-blind review were not based on well-calibrated beliefs, and there was no strong evidence to suggest that knowing the authors’ characteristics would improve peerreview. The results also suggest that double-blind review may also reduce the ability of reviewers to take into consideration when submissions are from under-represented groups. We discuss ways blind review may be adapted to create an equitable merit-based evaluation.