Research / BFI Working PaperMay 02, 2022

Measuring Knowledge

James Heckman, Jin Zhou

Empirical studies in the economics of education, the measurement of skill gaps across demographic groups, and the impacts of interventions on skill formation rely on psychometrically validated test scores that record the proportion of items correctly answered. Test scores are sometimes taken as measures of an invariant scale of human capital that can be compared over time and people. We show that for a prototypical test, invariance is violated. We use an unusually rich data set from an early childhood intervention program that measures knowledge of narrowly defined skills on essentially equivalent subsets of tasks. We examine if conventional, broadly-defined measures of skill are the same across people who are comparable on detailed knowledge measures. We reject the hypothesis of aggregate scale invariance and call into question the uncritical use of test scores in research on education and on skill formation. We compare different measures of skill and ability and reject the hypothesis of valid aggregate measures of skill.

Additional Materials

More Research From These Scholars

BFI Working Paper Feb 28, 2022

Causality and Econometrics

James Heckman, Rodrigo Pinto
Topics:  Uncategorized
BFI Working Paper Oct 4, 2022

Three Criteria for Evaluating Social Programs

James Heckman, Jorge Luis García
Topics:  Economic Mobility & Poverty
BFI Working Paper Dec 1, 2023

Econometric Causality: The Central Role of Thought Experiments

James Heckman, Rodrigo Pinto
Topics:  Uncategorized