So Rufus, as you have seen, left behind good records. He originally enlisted at Columbus, Mississippi so he may have moved there prior to the war. But then Columbus fits with your theory of the men travelling on the Tombigbee, since it was the farthest north navigable town on that river. Rufus was captured at Shelbyville, TN in 1863, sent to Nashville, then Louisville, then Camp Chase, then Fort Delaware where he died November 12, 1863 of an inflammation of the lungs. The records say he was buried on the Jersey shore.
I don't pick up any records for Benjamin Wilder. He could have served in a home guard unit and not left enough of a paper trail to get on official registers. I looked at both Alabama and Mississippi state records and didn't get a hit.
There is a James W. Wilder who enlisted in the 42nd Alabama Infantry at - again - Columbus, Mississippi. James enlisted May 7, 1862. He was paid in June by a cousin of mine! So funny to see that. He was captured and paroled at Vicksburg. I don't have anything else on him.