It's not so much swapping out as wearing out and replacement uniforms when they finally arrived tended to be from more standard stocks. That being said many units do seem to have tried where possible to retain at least parts of their distinctive uniforms. Furthermore as to being 'we make very bright and visible targets' I must admit I find this an overused argument for at the very minimum the coat of union blue was not exactly camouflaged. Furthermore they are carrying many items that glint and make a lot of noise so it is not as if a 'standard' unit of infantry is that likely to go unnoticed (well, unless there is no wind and the choking smoke from so much black powder refuses to lift. Though in that case it wouldn't matter what anyone was wearing either - though for differing reasons.)
As a final point although commonly called Zouaves and dressed either on uniforms loosely based on the Zouaves or the Chasseur de Afrique they were almost never trained in the correct Zouave tactics (Instead their training was based on the various so called Zouave drill companies that toured the States in the years before the war).