Since the ’80s the US has experienced both an increase in income inequality and an increase in residential segregation by income. After documenting this fact, we develop a general equilibrium model where parents choose the neighborhood where to raise their children. Segregation and inequality amplify each other because of local spillovers that affect the education returns. We calibrate the model using 1980 US data and the estimates for neighborhood exposure effects in Chetty and Hendren (2018b). We then show that segregation contributes to 28% of the increase in inequality between 1980 and 2010 after an unexpected permanent skill premium shock.