According to Spencer Tucker in the
Almanac of American Military History, Vol. 1, "bummers" had three meanings
1) Foraging or marauding soldiers in the Civil War. Union Army bummers were especially active during Major General William T. Sherman's March to the Sea
2) A person who was safely to the rear of the Army or otherwise away from combat
3) A forage of fatigue cap
#2 seems to match most closely with the question, but Tucker does not distinguish that this "bummers" were from a different branch of service.
According to
"War Slang: American Fighting Words & Phrases Since the Civil War", a bummer was a deserter or a predatory soldier, but eventually came to have a broader meaning to include "the destructive horde of deserters, stragglers, runaway slaves and marauders who helped make life miserable in the war-torn South." There were some specific types of bummers, such as Sherman's Bummers, who robbed pillaged and burned along with Sherman's army in Georgia; and
4) there were hospital bummers, who faked illness to get out of duty and into a military hospital (but again, this does not specify a separate branch of service)
.
source:
http://ironbrigader.com/2014/02/20/civil-war-slang-category-quiz-show-jeopardy/
and
The Language of the Civil War by John Wright (p.150)
FINAL ANSWER
5) A slang term used by the Navy (Union sailors) for a mortar boat
source:
The Language of the Civil War by John Wright (p. 46)
https://books.google.co.uk/books?id...TBRQ4ChDoAQg-MAY#v=onepage&q= bummer &f=false
For fun:
The term was not shortened to "bum" until around 1870. "War Slang" says "It is almost certainly a modification of the German Bummler (loafer)." The slang meaning of bummer today began with the advent of bad trips from hallucinogenic drugs in the 1960s.While "bummer" was evolving into plain old bum, another term sprang up for former Civil War soldiers which differentiates them from the lazy loafer and common bum. "Weary Willies" were Civil War vets who wandered about the land, setting up makeshift shelters for themselves, existing on handouts. The term at least acknowledges that they had been through an ordeal. The Civil War connection got a bit lost when circus performer Emmett Kelley created his famous "Weary Willie" character during the Great Depression. But the term reconnected with the Civil War in a 1970 episode of the TV Western "Bonanza." The Cartwrights allowed a group of Weary Willies to set up camp on Ponderosa land but the townfolk weren't particularly happy about it.