Answer: Attributed to the U.S. Navy and it described mortar boats, mortar schooners, mortar fleet or the sailors who manned them.
Sources:
"Mortars mounted on more seaworthy schooners would also play a key role in the capture of the first major Southern city: New Orleans. Under command of the legendary Admiral David Farragut, the Union Navy bombarded, reduced, and captured two key bastions, Forts Jackson and St. Philip, some 70 miles downstream of New Orleans. Twenty miles upstream from the Head of the Passes, the two forts were the keys to capturing the city, and Farragut used his own touches to do the job. Under the command of his stepbrother, Commodore David Dixon Porter, the mortar schooners (called 'bummers' by their crews) were brought into position and began to bombard the two forts on April 18, 1862."
Civil War Artillery
"The active part he took in the reduction of the forts below New Orleans will make his name forever memorable in connection with the mortar fleet, or 'bummers,' as the sailors term them."
David D. Porter
"Bummers" is a term frequently used pejoratively to denote malingerers, men who hang behind in order to scavenge. For sailors of the inland navy, however, "bummers" was a slang term for the mortar schooners and their crews, presumably a variant on "bombers."
Two Civil Wars: The Curious Shared Journal of a Baton Rouge Schoolgirl and a Union Sailor on the USS Essex, edited by Katherine Bentley Jeffrey
Edit - Players came us with a lot of different definitions for "bummer" and as far as I can tell, they were correct in stating that the term was used in the ways they described. However, I am only going to accept the official answer as being correct for this question.
The key to the question is the fact that, after mentioning the fact that the term "bummer" was used to describe marauding or foraging soldiers of Sherman's army, the question then made it clear that the desired response should relate to the use of that term in connection with another branch of service - i.e., a branch of service other than the Army.
That essentially rules out all the answers that described "bummers" as being associated with the Army. Note that neither the medical corps nor hospital corps nor ambulance corps are considered to be a separate branch of service - they are part of the Army or Navy or whichever branch to which they belong.
Hoosier