Economists have long argued that innovation is an essential driver of economic growth, with some estimates suggesting that roughly 50 percent of US annual GDP growth is attributable to innovation. Likewise, policymakers have long paid particular attention to stimulating innovation and to the supply of new technologies, while economists have studied both pecuniary and non-pecuniary aspects of technology adoption.
However, innovation alone cannot drive growth; users must also adopt new technologies. Likewise, an effective innovation is not measured by its potential returns but, rather, on its effective returns to scale, and “scale” is the operative word driving recent research. For example, research questions revolve around whether small-scale research findings persist in larger markets and broader settings. Further, what happens when interventions are scaled to larger populations? Should we expect the same level of efficacy observed in the small-scale setting? If not, then what are the important threats to scalability? More than an academic exercise, a proper understanding of these and related questions can avoid wasted resources, improve people’s lives, and build trust in the scientific method’s ability to contribute to policymaking.
This work explores the scale-up problem for an important class of new technologies in the energy space—thermostats that leverage smart functionalities and, thus, hold up the promise of more efficient energy use. The authors examine data from two framed field experiments, wherein the 1,385 households that volunteered to participate in the study were randomized into either a treatment group that received free installation of a two-way programmable smart thermostat, or a control group that kept their existing thermostat. The authors analyze energy consumption over an 18-month period that includes more than 16 million hourly electricity use records and almost 700 thousand daily observations of natural gas consumption, to find the following:
- Smart thermostats have neither a statistically nor economically significant effect on energy use. Indeed, some estimates suggest smart thermostats may actually increase electricity and gas consumption by 2.3% and 4.2%, respectively. These results mirror a growing body of research on the real-world effects of “energy efficient” technology.
- Smart thermostats under-deliver on the savings promised by engineers. By employing a model that better incorporates human adaptation to the technology adaptation, and checking that model against higher-frequency data, the authors can investigate whether this aggregate result masks significant, but offsetting, heterogeneous effects that may have implications for how the intervention scales to different settings. The answer is that there is almost no evidence of heterogeneous treatment effects.
- Why do smart thermostats fail to scale from the engineer’s lab to the household’s wall? Because users frequently override permanently scheduled temperature setpoints, and those override settings are less energy efficient than the previously scheduled setpoint. This finding is based on the authors’ analysis of nearly 4 million observations of treatment group heating, ventilation, and air conditioning (HVAC) system activity and user interactions with their smart thermostat in the form of scheduled temperature setpoints, temporary overrides, and HVAC system events.
- Finally, having categorized smart thermostat households into how intensively they use the energy-saving features of their thermostat, the authors find that while some user types realize significant savings, engineering models fail to capture how most people actually use smart technologies, thus limiting the usefulness of their estimates in real-world settings. In other words, while people may adopt smart technology, most use its features in ways that undo purported benefits, suggesting that human behavior is a peril to scaling such technologies.
For policymakers—and researchers—this micro example has a macro bottom line: Projected savings from innovations that fail to account for how people use new technology are often overly optimistic and potentially costly. Innovation for its own sake will not spur economic growth and improve quality of life; users must adapt, and assumptions on user uptake need reality checks.