The problem is you're just accepting whatever Calhoun says at face value. Even when he's being relatively honest, he's engaged in a heavy dose of obfuscation. What Southern interests besides slavery did Calhoun specifically refer to? The purpose of some vague reference to all their other interests is rather obvious. And its the same purpose that the positive good slavery theory served. Slavery, a recognized evil, would be very difficult to defend. Which is what nullification and secession 1828-1832 were actually focused on. Set a precedent with a tariff, which is easier to attack on moral grounds than slavery, and which as we know Calhoun himself had helped make as abominable as possible, and it's there if and when you need it to preserve slavery. Why was he originally a proponent of the American System? This is the guy now arguing that protectionism is unconstitutional. And why did he flip when he did? Just as abolitionism was becoming radical. And the fact is the states had not reserved the rights Calhoun supposed they had. How could the states reserve unilateral powers over a supreme national government that did not previously exist? If they thought they had constitutional arguments, they would have pursued them via the courts.The point that Calhoun was making was that if the federal government could use the threat of force to make the Southern States abide by a tax policy they saw as fundamentally against their interests, that all of their interests were therefore under threat if the rest of the Union was against any of them, including slavery. The tariff complaints and problems were very real, with the conflict over the tariff making it clear that there were wider implications if the reserved rights of the States could not protect them from a majority in the Union.
Not only is Calhoun's concern slavery, and only slavery, but slavery was not directly threatened by tariffs/taxation. The population of slaves in the US steadily progressed (Source), AND the average value of slaves more than doubled (Source), from 1828 to 1838 when tariffs were at their highest (Source).