Have you ever tested multiple subject lines on an email to see which got more opens? Cooked a recipe two ways to settle a debate at home? If so, you’ve dabbled in the ideas that John List has spent more than three decades championing. You’ve run an experiment!
The instinct to perturb the real world, see what happens, and learn from it is the central premise of List’s new textbook, Experimental Economics: Theory and Practice (University of Chicago Press). At nearly 900 pages, the book is the most comprehensive treatment to date of the experimental methods List has pioneered, covering the design, ethics, analysis, and scaling of field experiments. It includes experiments spanning from lab experiments to natural field experiments, which List is credited with innovating in the early 1990s.
Why Experiments?
Prior to the rise of experiments, economists typically stuck to the questions they could answer with available data collected through observing the world around us. While these observational data are great for describing patterns — identifying the what — they are limited for uncovering the why. Earnings data, for example, reveal that women earn less than men for the same work, but not what produces the gap.
The challenge of causation is one List often illustrates with a childhood memory. “My mom was convinced that if you ate ice cream, you needed to rest on the couch for 45 minutes before you went swimming,” he recalled on a recent episode of The Pie podcast. Data show that ice cream consumption and drowning rates tend to rise at around the same times, convincing List’s mother that swimming after ice cream was dangerous. The real culprit, of course, was heat, driving the rise in both.
The world is rife with such confounders, which are nearly impossible to capture with even the most exhaustive dataset. The gender pay gap might reflect gender bias, or it could reflect differences in factors like experience or negotiation tactics between men and women. Attributing causation to differences observed between groups is nearly impossible due to the many factors that differ between them.
Life as Lab
The traditional gold standard for establishing cause and effect is the randomized controlled trial, or RCT for short. Common in medicine, an RCT randomly assigns participants to a treatment group and a control group, then compares outcomes. But economists had long considered this kind of clean experimentation off-limits in their own discipline. When List opened his first economics textbook in 1987, it told him as much. The world, the textbook explained, was too messy, too full of confounding variables, for the RCT logic to apply outside the lab.
List disagreed. “Randomization does not get rid of the dirt,” he said on UChicago’s Big Brains podcast in 2019. “The beauty behind randomization is when you randomly put some people in treatment and some people in control… it balances the dirt across the treatment and control groups… and now I can learn something causal in the real world.”
With that, List was off to the races. In the decades since, his experiments have spanned schools, charities, governments, and firms — each one designed to generate the kind of causal evidence the world wasn’t producing on its own. To study what improves outcomes for young children, for example, List launched a preschool in Chicago Heights in 2010, randomly assigning children to different curricula and tracking results over years. To study charitable giving, he has worked with organizations like Smile Train, which sends roughly a million fundraising solicitations a month, randomizing the wording of letters across donor groups.
Lessons from Three Decades of Experiments
The book is built around examples like these, drawn from more than three decades of work in positions including chief economist at Uber, Lyft, and Walmart, and senior economist on the White House Council of Economic Advisers under President George W. Bush. It is also, by List’s own description, a catalogue of his own missteps. “I have made enough experimental mistakes to fill a large tome,” he writes, and many of the worked examples are studies he himself published, annotated with the design choices he would make differently today. It is the field guide he wishes he’d had as a young researcher.
A central concern of the book is what happens after the experiment ends and a promising result has to scale. “Can your idea change the world,” List often asks. This is the “voltage effect” List explored in his previous book of the same name: the tendency for findings to lose much of their punch when expanded from a small pilot to a broader population. The Chicago Heights preschool is a useful illustration. A program staffed by 30 carefully recruited teachers might post stunning results in a single neighborhood, but those results say little about what will happen when a district has to hire 3,000. Experimental Economics presses researchers to design with that future in mind from the very first study.That means deliberately recruiting an average mix of teachers, sites, or participants, checking whether the program’s inputs can be procured at scale, and asking up front which features of a result are likely to travel and which are artifacts of the original setting.
Aimed primarily at PhD students, master’s students, and advanced undergraduates, Experimental Economics is also being adapted into a shorter, math-light undergraduate edition titled Everyday Experimentation. “Use your life as a lab,” List told listeners of The Pie. “Every time you make a decision, try to design that decision so you can learn something.” The book is written in that spirit — a guide for more people to change the world by testing what works and why.
List’s motivation is also personal. He traces his approach back to a folding card table in Wisconsin, where as a teenager he sold and traded baseball cards on weekends and began informally testing how different bargaining tactics worked on different kinds of buyers. He didn’t know it at the time, but he was already running natural field experiments — the same kind that, decades later, would reshape how policymakers and firms think about the value of time, charitable giving, discrimination, and early childhood education. His hope is that somewhere out there is another kid at another card table — or behind a lemonade stand, or a garage sale, or a school fundraiser — quietly running their own experiments and beginning to wonder whether the textbooks have it right. Experimental Economics is, in part, written for them.
View the book