I would say its because outside of a few State Militia units, and the C.S.N., Blacks weren't allowed to enlist in the Confederate military. But that doesn't mean there weren't Blacks in Confederate uniform, in Texas there was a labor bureau filled with slaves, uniformed by the Confederate government, building fortifications and such in uniforms made for them at the Houston Depot. As for Blacks in the field, armed and equipped as soldiers, I'd say its because through any number of ways they were with the Army, and occasionally put to work as pickets, guarding prisoners and such by either their masters, or by officers above their masters using all the slaves in camp for some purpose, or they were part of a group of slaves attached to the Army "on loan" and simply ended up armed, and someone saw them and assumed they were enlisted personnel, because they looked the part, and in some cases performed the part.
Now is that to say legions of loyal Blacks were loyally serving? No. But does that say they all were apart of it against their will? Not really, post-war photos of Black attending Confederate reunions seem to show some were evidently proud of their service. It all comes down to the individual views and experiences on that count.
The Confederacy officially didn't allow Blacks in the Army till the end of the War, but through many ways Blacks ended up doing soldier-like duties in uniforms, as laborers and servants, and apparently more than a few ended up on the battlefield or near it looking the part as soldiers, and it seems a lot of them were proud of it later in life, so it can be said Black Confederate "soldiers" existed, even if they weren't officially soldiers.