On the contrary, nationwide there would have been almost no consequences for a "failure" to serve. You're conflating today's values with those of the 1860's. In any city, town, village in the US there would have been hundreds of men working at commonplace jobs compared to those who volunteered or were drafted. Plus America had not become a middle class dominated society yet. The people who could afford to pay a substitute would simply be exercising the US equivalent of noblesse oblige. Can you imagine a common farmer or tradesman/factory worker in Europe criticizing the local lord, baron, earl, duke, count, etc. for ANYTHING?
You would expect the Confederate armies would have been wall to wall plantation owners, especially given the basic thing the Confederate armies were fighting to protect--slavery. To the contrary, very few were like Wade Hampton who served with great skill. The plantation owners had to stay at home and run the plantations.
I know you have seen/heard constant references to the "greatest generation" who fought in WWII. And believe me I am not criticizing them. But the overwhelming majority of men who served in WWII were not volunteers they were drafted. It was not until Vietnam reached it's apex that all men became eligible to the draft. I remember vividly being in college and catching the newspaper in mid air the morning the draft numbers were first printed. (I was #344 so I'm still waiting for WW 13 to be called to serve.) Even in 1968, although the concept was beginning to change, the basic concept of college was that it was reserved for middle and upper classes so they could learn how to run the country, while those "lesser" people bled and died.