We examine the social construction of race during the United States’ Reconstruction Era, a critical juncture between slavery and Jim Crow segregation. We show that people with the same detailed skin tone, recorded by the Freedman’s Bank (1865-1874), were more likely racialized as White or Mulatto by the 1870 Census if they were wealthier or literate. Our estimates reveal the construction – or rather, reconstruction – of race in a period of unfulfilled potential for social transformation, setting a path for racial segregation and continued racial stratification. The endogenous historical construction of race also has implications for analyses that compare individuals by race or include race as a control variable.