Howdy and Welcome!
When Sherman was at present day LSU (pre-war), he loved the South and the people in it. I may be wrong, but I find it difficult to believe that he would have stereotyped Southerners by describing them as people "who knew nothing but hunting, drinking, gambling and dueling, a class who benefited from slavery and would rather die than work for a living."
This description sounds more like an episode of "Hee Haw" or "The Dukes of Hazzard"....
Welcome, from Alabama! I cannot help with your exact quote attributed to Sherman, but the sentiment is clearly echoed throughout his memoir, and other of his similar comments, and in personal letters. It wasn't just the South that Sherman hated, it was also Native Americans, nor did he respect the lives of blacks. You can find much about Sherman in various posts herein.