I think that the white population increased due to natural increase and immigration. The enslaved grew due to natural reproduction. The fact that the free non-white population grew at its lowest rate in the decade before the war, at half the rate of enslaved blacks, indicates lack of a transition to free labor.Or could the differences be found somewhere in the immigration stats..?
The 31% WP could easily be the result of natural increase, plus immigration. The 23% slave increase is easily deduced from natural reproduction. How many "free non-whites" immigrated to the US 1850-1860..? How about 1790-1800..?