It helps to recall what Grant's specific orders were: "You will, no doubt, clean the country where you go of railroad tracks and supplies. I would also move every wagon, horse, mule, and hoof of stock, as well as the negroes.”
The valuations placed on slaves as chattel property are not to be trusted. There was regular trading in the sugar and cotton that slaves produced, with futures contracts and debt securitizations that reached across the Atlantic. The shares and bonds of Southern railroads traded in New York, Philadelphia and London. Southern plantation land could be mortgaged and sold. But, for the asset that was supposedly worth more than everything else in the United States of America, there were no futures contracts, no debt securitizations and no central financial markets here or abroad.
The premise that the slaves by themselves were an immense store of wealth is based largely on the records generated from the buyout by Parliament of the English owners of slaves throughout the empire in 1837. In the U.S. a quarter century later we have only the records of a few local slave brokerages; for Britain we have the full accounts of 40,000 claims and the price paid for each. The total payments were 40% of the Empire's official budget for that year.
https://archive.org/details/ukslaveownercompensation/page/n1/mode/2up
What slaves produced was worth a great deal of money and that gave real value to plantation real estate. What slaves themselves were worth as chattel property, independent of the plantations that they worked on, was a question no serious speculator wanted to bet on.
The conduct of the slave owners themselves is the tell. The promising cotton and sugar lands opened up in the 1820s and 1830s by the removal of the Cherokees and others were not settled by producers who bought the land and then imported slaves; they were pioneered by growers who moved their entire operations, including their slave families with them.
The threat of ending slavery in the territories was a real economic threat precisely because new lands gained their value from settlement by slave owners. The slaves alone could not be sold at any profit. The lands, once established as plantations, would have immense value that could be mortgaged and sold.