In Battle Cry of Freedom, James McPherson writes that everyone who could manage it left town. There was high demand for carts, wagons, and any horse or mule that could walk. The population of Richmond had swelled though, after it became the capitol of the Confederacy. With the government gone, no doubt there were people who never came back because their jobs had disappeared. The estimate I find on the web suggests the city grew from @ 38,000 people (11,739 enslaved) to as many as 150,000 by 1865. Surely a good portion of the population could see the handwriting on the wall and left before the first of April?