Inference for Ranks with Applications to Mobility Across Neighborhoods and Academic Achievement Across Countries

This idea has resonance beyond the ivory tower and has entered into policy discussions at various levels of government, with officials encouraged to use these new mobility numbers to assess the efficacy of anti-poverty and other income-enhancing programs at a local level,¹ and with all policymakers and others urged to compare mobility rates across the United States through the creation of an online atlas.²  

While this research has generated important insights into the phenomenon of intergenerational mobility across broad regions of the United States, how confident can we be about findings that make assessments about such mobility across counties and cities within those regions, and especially at a highly granular level like neighborhoods within those cities? In their new paper, “Inference for Ranks with Applications to Mobility Across Neighborhoods and Academic Achievement Across Countries,” UChicago’s Magne Mogstad and Azeem M. Shaikh, along with Joseph P. Romano of Stanford and Daniel Wilhelm of University College in London, address this question and others to find that distinctions about income mobility that exist at a macro level tend to fall apart when viewed more closely. These results have important implications for policymakers charged, for example, with applying limited resources to programs that improve opportunities for low-income families. 

Large lists + small places = much uncertainty 

At the core of this work is an insight regarding the nature of rankings among places. That insight is simple in description but powerful in practice: When you rank 10 cities on a particular scale, for example, you are saying that No. 1 is better than all the others, No. 2 is worse than No. 1 but better than Nos. 3-10, and so on. To develop your ranking, you have to make comparisons across those 10 cities—each with the other nine. However, with each comparison there is a level of statistical uncertainty. The more comparisons you make, the more uncertainty you create. Now imagine a list of 100 places, or 1,000, with its many thousands of comparisons. How certain can you be that No. 375 is actually better than 425? Or even whether No. 100 is better than 900? 

Putting this insight into empirical practice, the authors construct detailed confidence sets that allow them to account for uncertainty within rankings, and they begin by applying their methodology to an analysis of the Programme for International Student Assessment (PISA)³, an organization that ranks different populations according to academic achievement. To summarize the authors’ exhaustive approach, (please see the full paper for details), their analysis of PISA’s rankings finds that, indeed, it is possible to establish a ranking of how countries perform in regards to reading, math, and sciences that is robust to accounting for uncertainty. The authors offer this example to show that not all rankings are created equal: Researchers can adequately account for uncertainty and still post rankings with an appropriate degree of confidence. 

This new research finds that Chetty et al.’s rankings are most robust for uncertainty when they are restricted to the 50 most populous commuting zones (CZs) or counties in the United States. Also, this research confirms Chetty et al.’s findings that, broadly speaking, upward mobility is low in areas of the Southeast and high in areas of the Great Plains.

Next, the authors turn their attention to the claims made about intergenerational mobility in papers authored by Harvard economist Raj Chetty et al. (2014, 2018) and Chetty and Harvard’s Nathaniel Hendren (2018). (For the purposes of this Research Brief, “Chetty et al.” will refer to this entire scope of research; please see full paper for citations.) As noted above, this work has been embraced by many economists and policymakers, inspiring new research programs and policy proposals. Mogstad, Shaikh, Romano, and Wilhelm find that Chetty et al.’s rankings are most robust for uncertainty when they are restricted to the 50 most populous commuting zones (CZs) or counties in the United States. Also, this new research confirms Chetty et al.’s findings that, broadly speaking, upward mobility is low in areas of the Southeast and high in areas of the Great Plains. 

However, Mogstand, Shaikh, et al.’s extensive analysis also reveals important facts that run counter to Chetty et al., which are highlighted here:

• Conclusions about geographic patterns of income mobility become less robust when all commuting zones are considered.
• The West Coast and Northeast have relatively low mobility.
• Within-region variation is not evident.
• Importantly, when the authors refine the data to the county level (counties are often smaller than CZs), the rankings are largely uninformative; there is simply too much uncertainty at more granular levels. 

Figure 1 illustrates the broad conclusions of Chetty et al. that are robust to accounting for uncertainty when considered at the commuting zone level, while Figure 2 shows how uncertainties increase at the county level and, thus, run counter to many of Chetty et al.’s conclusions about place and intergenerational mobility. 

The fourth point listed above—that claims about mobility do not hold at granular levels—is key to this new research’s critique of Chetty et al. Likewise, to further investigate this question, the authors apply their methodology to a review of Creating Moves to Opportunity (CMTO), a housing mobility program in Seattle and King County that aims to reduce rental barriers and move eligible families to “opportunity neighborhoods.”⁴ In many respects, CMTO is the policy embodiment of Chetty et al.’s research program. In this case, a treatment group of low-income families was offered financial support and assistance to find and lease units in areas classified as high upward-mobility neighborhoods within the county, or those in the top third as defined by CMTO. 

However, those definitions are spurious, according to Mogstad, Shaikh, et al., who show that such upward-mobility neighborhoods do not have statistically higher mobility rates than other neighborhoods. Rather, CMTO’s description of neighborhoods as containing either high or low mobility rates likely results from statistical uncertainty of the nature described above. Indeed, the authors find that all but two of the 397 census tracts⁵ could realistically be considered as among the top third in terms of income mobility. 

CLOSING TAKEAWAY
In a review of Creating Moves to Opportunity (CMTO), a housing mobility program in Seattle and King County that aims to reduce rental barriers and move eligible families to “opportunity neighborhoods,” the authors show that so-called upward-mobility neighborhoods do not have statistically higher mobility rates than other neighborhoods. Rather, CMTO’s description of neighborhoods as containing either high or low mobility rates likely results from statistical uncertainty. Indeed, the authors find that all but two of the 397 census tracts could realistically be considered as among the top third in terms of income mobility.

Conclusion and Implications

Income inequality, income mobility over generations, and income mobility across space are currently of primary interest among researchers and policymakers. Units of government, from federal to local, have limited resources to address these issues and caution should be taken when devising policies that commit those resources to programs that over-promise on delivery. In the case of income mobility and the research/policy programs inspired by Chetty et al., the useful insights about the broad variation in mobility that exist among US regions have been lost in the attempt to extend those findings to a granular level. What is true of geographic regions is not necessarily true of neighborhoods.

In 2014, Chetty testified before Congress and encouraged policymakers to focus on particular cities that fall on the lower end of mobility rankings, rather than devising policies on a national scope, since such local policies would prove most effective. Further, he advocated for officials to publicize mobility rankings in the hopes that drawing attention to “areas that need improvement” would motivate action among local policymakers.⁶ As described above, and as illustrated in greater detail in their working paper, the authors find that such confidence in city and neighborhood rankings is misplaced. Likewise, policies meant to move people from presumed low-mobility neighborhoods to high-mobility neighborhoods are often simply moving people around to little or no effect.

¹ Chetty, R. (April 1, 2014). Improving opportunities for economic mobility in the United States. Budget Committee United States Senate.

² opportunityatlas.org

³ oecd.org/pisa

⁴ opportunityinsights.org/policy/cmto

⁵ Census tracts are small, relatively permanent political subdivisions of a county, averaging about 4,000 inhabitants.

⁶ Chetty, op. cit.