It is my opinion that these men are from Co. I of the 55th New York Infantry "The Lafayette Guard". The 55th was formed from the 55th NY State Militia and was predominately comprised of French immigrants from New York City. Regis de Trobriand was responsible for the unit's organization. The militia unit being comprised mostly of Frenchmen, adopted a uniform that reflected French army uniforms complete with dark blue frock coats with red trim, shakos, and red trousers.
http://appliedvb.com/NY40th/FRENCH.html Sometime later, Company I was formed as a Zouave unit.
http://dmna.ny.gov/historic/reghist/civil/infantry/55thInf/55thInfCWN.htm
The men in the photograph are clearly Zouave troops. If you notice, at least four definitely have shakos on. The three wearing havelocks might also be wearing shakos underneath. I suspect this because of how high they sit on the men's heads, versus how high a kepi or forage cap would normally sit. One private wears a forage cap. The officer wears a kepi and trousers. (typical in a Zouave unit) The soldier standing on the parapet is wearing a fez. Because a number of the men are wearing shakos, and the original 55th wore shakos, I would argue that it is possible a number of the original members of the 55th Militia retained the shakos out of a sense of pride.
The 55th New York were in action at Yorktown during the Peninsula Campaign, 1862.
http://www.civilwarintheeast.com/USA/NY/NY055.php
These men and their uniforms definitely have "foreign" look to them. In fact, when I was first presented with this photograph, I first thought it was of French troops in the Crimea. It was only when I saw the calk identifier on the gun carriage pointed out above did I realize it was CW. So, to try and discover who these men might be, I started searching foreign units serving I the federal army, then narrowed that to Zouave units, then those that were on the Peninsula. This is how I came up with the 55th NYVI.