First off, it was 180,000 troops and not 100,000. But even 180,000 is still not that much compared to 2.8 million, as you pointed out. But I think you need to look at the quality of the service of the Black Civil War soldier moreso than the quantity.
When the Emancipation Proclamation went into effect (long before the Federal soldier draft), it clarified once and for all that Blacks would be enlisted into the Army. Northern Blacks would no longer be turned away; and men escaping from slavery would have the opportunity for military service and combat against their former oppressors. Debate why the war was fought if you want to but there is no question why the USCT soldier fought the war.
I think their greatest impact was psychological. USCT soldiers certainly made a difference to the Confederacy, who chose to not treat them as soldiers at all; instead, murdering them when they tried to surrender after the Union defeats of Olustee, Fort Pillow Poison Springs and the Crater. But to be fair, over 100 soldiers were treated as POWs after Olustee and sent to Andersonville.
But if you want to talk about quantity, remember that 180,000 soldiers is the size of two Civil War armies. And to the 900,000-1,000,000 who fought for the Confederacy, 180,000 is not that small of a number.