Research / BFI Working PaperMay 20, 2019

The Science of Using Science: Towards an Understanding of the Threats to Scaling Experiments

John List, Dana Suskind, Omar Al-Ubaydli

Policymakers are increasingly turning to insights gained from the experimental method as a means of informing public policies. Whether—and to what extent—insights from a research study scale to the level of the broader public is, in many situations, based on blind faith. This scale-up problem can lead to a vast waste of resources, a missed opportunity to improve people’s lives, and a diminution in the public’s trust in the scientific method’s ability to contribute to policymaking. This study provides a theoretical lens to deepen our understanding of the science of how to use science. Through a simple model, we highlight three elements of the scale-up problem: (1) when does evidence become actionable (appropriate statistical inference); (2) properties of the population; and (3) properties of the situation. We argue that until these three areas are fully understood and recognized by researchers and policymakers, the threats to scalability will render any scaling exercise as particularly vulnerable. In this way, our work represents a challenge to empiricists to estimate the nature and extent of how important the various threats to scalability are in practice, and to implement those in their original research.

More Research From These Scholars

BFI Working Paper Aug 19, 2020

The $100 Million Nudge: Increasing Tax Compliance of Businesses and the Self-Employed using a Natural Field Experiment

Justin E. Holz, John List, Alejandro Zentner, Marvin Cardoza, Joaquin Zentner
Topics:  Fiscal Studies
BFI Working Paper Apr 30, 2019

Measuring Success in Education: The Role of Effort on the Test Itself

John List, Uri Gneezy, Jeffrey A. Livingston, Xiangdong Qin, Sally Sadoff, Yang Xu
Topics:  Early Childhood Education
BFI Working Paper Dec 28, 2020

The Social Side of Early Human Capital Formation: Using a Field Experiment to Estimate the Causal Impact of Neighborhoods

John List, Fatemeh Momeni, Yves Zenou
Topics:  Early Childhood Education