Research / BFI Working PaperMay 02, 2022

Measuring Knowledge

James Heckman, Jin Zhou

Empirical studies in the economics of education, the measurement of skill gaps across demographic groups, and the impacts of interventions on skill formation rely on psychometrically validated test scores that record the proportion of items correctly answered. Test scores are sometimes taken as measures of an invariant scale of human capital that can be compared over time and people. We show that for a prototypical test, invariance is violated. We use an unusually rich data set from an early childhood intervention program that measures knowledge of narrowly defined skills on essentially equivalent subsets of tasks. We examine if conventional, broadly-defined measures of skill are the same across people who are comparable on detailed knowledge measures. We reject the hypothesis of aggregate scale invariance and call into question the uncritical use of test scores in research on education and on skill formation. We compare different measures of skill and ability and reject the hypothesis of valid aggregate measures of skill.

Additional Materials

More Research From These Scholars

BFI Working Paper Jul 15, 2021

The Lasting Effects of Early Childhood Education on Promoting the Skills and Social Mobility of Disadvantaged African Americans

Jorge Luis García, James Heckman, Victor Ronda
Topics:  Early Childhood Education
BFI Working Paper Jun 30, 2021

The Dynastic Benefits of Early Childhood Education

Jorge Luis García, Frederik H. Bennhoff, Duncan Ermini Leaf, James Heckman
Topics:  Early Childhood Education
BFI Working Paper Jul 16, 2020

Inference with Imperfect Randomization: The Case of the Perry Preschool Program

James Heckman, Rodrigo Pinto, Azeem Shaikh
Topics:  Early Childhood Education