It wasn't 'imaginary' - they carried it on a pole and held it up so everyone could see. Yes, it was a great honor, but also a great target.
The colors were a rallying point in battle - no radios - remove that and Pvt. Joe could not have a visible rallying point when times got tough - and the artillery lost sight of your unit too! If the colors went down in attack or defense, the gunners tended to be told to target the area to ensure the enemy did not have it easy!
One of Terry's Texas Rangers refused to be a color bearer just because of that reason! He kept telling them, "I'm not the man for that job." And he never became one either!