In terms of volume, the cotton growers were doing fine. But the British and French had saturated the world market with factory cloth. The price cotton was getting noticeably soft, just as the prospects for the unmodified continuation of slavery in the US began to look doubtful.Olmstead's descriptions of the backward Carolinas can easily be matched by the complaints and scorn that English visitors had for the condition of the Irish in Boston and through New England during the same period. Against his abolitionist propaganda one has to balance the facts of what was actually happening economically.
Saying that the planters were broke does not make it so. The cotton South was the one part of the country that did not suffer during the Panic of 1857. They had completely recovered from their period of financial crisis - the 1840s. The trade in cotton was booming in volume and price. Planters could not abandon the political argument that the South was much put upon; they had been using that theme for a quarter century. But, their economic view in 1860 was optimistic, not fearful; they argued in favor of Southern slavery's expansion to the territories and overseas because they anticipated making more money, not because they were afraid of losing what they had.
As far as territorial expansion of cotton production, it happened in the post war era. Texas became a major producer of cotton, and the effect on the price of cotton and per capita income in the southern areas was negative.