We explore the impact of rising incomes at the top of the distribution on spatial sorting patterns within large U.S. cities. We develop and quantify a spatial model of a city with heterogeneous agents and nonhomothetic preferences for locations with different amenities of endogenous quality. As the rich get richer, their increased demand for luxury amenities available downtown drives housing prices up in downtown areas. The poor are made worse off, either being displaced or paying higher rents for amenities that they do not value as much. Endogenous provision of private amenities amplifies the mechanism, while public provision of other amenities in part curbs it. We quantify the corresponding impact on well-being inequality. Through the lens of the quantified model, the change in the income distribution between 1990 and 2014 led to neighborhood change and spatial resorting within urban areas that increased the welfare of richer households relative to that of poorer households by an additional 1.7 percentage points on top of their differential income growth.