Surveys are a key tool for empirical work in economics and other social sciences to examine human behavior, while government operations rely on household surveys as a main source of data used to produce official statistics, including unemployment, poverty, and health insurance coverage rates. Unfortunately, survey data have been found to contain errors in a wide range of settings. For US household surveys, the quality of survey data has been declining steadily in recent years, with households more reluctant to participate in surveys, and participants more likely to refuse to answer questions and to give inaccurate responses.
Even though its relevance has been documented by researchers over the past two decades, there is still much to learn about measurement error, or how the reported responses of households differ from true values. In this paper, the authors study measurement error in surveys and analyze theories of its nature to improve the accuracy of survey data and estimates derived from it. The authors study measurement error in reports of participation in government programs by linking the surveys to administrative records, arguing that such data linkage can provide the required measure of truth if the data sources and linkage are sufficiently accurate. In other words, the authors link multiple survey results and program data to provide a novel, and powerful, examination of survey error.
Specifically, the authors focus on two types of errors in binary variables: false negative responses (failures of true recipients to report) and false positive responses (reported receipt by those who are not in the administrative data). Their findings, including the following, confirm several theories of cognitive factors that can lead to survey misreporting:
- Recall is an important source of response errors. Longer recall periods increase the probability that households fail to report program receipt. Problems of accurately recalling the timing of receipt, known as telescoping, are an important reason for overreporting.
- Salience of the topic improves the quality of the answer. The authors provide evidence that respondents sometimes misreport when the true answer is likely known to them, and that stigma, indeed, reduces reporting of program receipt.
- Cooperativeness affects the accuracy of responses, insofar as interviewees who frequently non-respond are more likely to misreport than other interviewees.
- Finally, regarding survey design, the authors find no loss of accuracy from proxy interviews. Their results on survey mode effects are in line with the trade-off between non-response and accuracy found in the previous literature.
This work has implications beyond the case of government transfers and the specific surveys studied in this paper and may allow data users to gauge the prevalence of errors in their data and to select more reliable measures. Further, the authors’ results and recommendations are broad enough to apply in many settings where misreporting is a problem. For instance, similar issues of data quality have been found in health, crime, or earnings studies, to name a few.