BG Frank Armstrong fought for the US at First Manassas, then resigned and took a commission in the CS in August 1861.
James Deshler technically never resigned from US service but rather was dropped for overstaying leave in July 1861.
Edit - Good answer. Though Deshler didn't necessarily fight for the US, he was in US service at the time the Civil War began.
Hoosier