Traditionally, the color-bearers were armed only with swords. It was an 'honor' sword and not intended to be used, even if it was just an issue sword. In many armies they were officers who wore their side arms only. They carried the regimental colors, usually two flags, the regiment's - unique to the regiment- and the country's/monarch's/state's - the more standard National Flag. That was a two-handed task. The traditional 'Color' is not 'just' a flag, more like a woven tapestry and very heavy, so they had no chance of wielding a weapon.
View attachment 538273
A sergeant - the National Color-bearer of the 108th US Colored Infantry. He is wearing a sword and a flag-holder cross belt.
They WERE the regiment and capture of either of the regimental colors was a foul disgrace. They were there to keep the honor of the regiment and to provide a rallying point for the men of that regiment, not to kill the enemy. They could not even defend themselves, hence the color-guard - two senior NCOs who were their personal bodyguard and a platoon or company from the reserve, just behind them. Their position was in the center of the line in battle; in front of the regiment on parade or after the leading company on the march, colors cased (wound carefully around the staff and covered to protect them).
The Civil War was a conflict which saw many traditions overturned so it was not only officers who carried the colors, it depended on who was available - and who was brave enough to become an unarmed target when it was needed.