My ancestor mustered in to the 16th NY Infantry in August, 1862. This was a two year regiment whose term of enlistment ended in May, 1863. In that month he was captured at the Battle of Salem Church, held briefly at Richmond, exchanged, and paroled. Apparently he was expected to transfer to the 121st NY with the other "three year men" because he was soon arrested for desertion (and released) by the Provost Marshal on the streets of his home town in upstate NY. At various times GG Grandfather claimed his enlistment was for two years (not true) and that he was at home "awaiting orders" (that never came.) In February of 1864 he joined the 73rd OH and served in that unit until war's end. The fact that he traveled way out west to join another unit rather than show up at the 121st NY suggests he was maybe a little guilty and afraid. What would his comrades say? What would they do to him?
These events caused some confusion when he applied for a pension but it was finally resolved in his favor.
Did he desert? I think he was a just a homesick boy who saw an opportunity to take some time off from the army and the war but eventually did the right thing. Either way, I certainly wouldn't judge him.