The supply of Richmond and the ANV was only barely possible as early as the spring of 1863. Lee was screaming (in his gentlemanly way) about the need for more food for his horses starting in November 1862. To help reduce his demand, he reduced the number of wagons allowed to the ANV and did not increase his artillery as much as he might have wished. He also sent Longstreet to cover the removal of supplies from the Suffolk area of Virginia, reducing the demand on the railroad supply line.
The impact of army operations on agricultural areas was sever. Forage operations, by both armies, stripped the fields and forced the farmers to leave the area to prevent starvation. Farms just south of the area of operations (ie southern Virginia and North Carolina) were far less productive than usual because of the very high percentage of the men in those 2 states who were in the army. So supplies had to come from South Carolina (but they used most of what they produced) and southwestern Georgia -- untouched by armies throughout the war.
Southern railroads were never designed to send goods from SW Georgia to Virginia, and the requirement to do so, while meeting all the other transportation requirements, was more than could be accomplished. Cavalry raids and army movements made things worse, but the wear and tear of the war's demands and the inability of the South to replace the iron, locomotives and cars as they were lost (to say nothing of expanding the stock) was a killer by itself. With the blockade preventing shipping by water and the manpower requirements preventing the expansion of the RR industry, they system was guaranteed to fail just about the time it did.