Okay, I think I've got it. All evidence suggests our Captain is Henry Clay Price of Missouri (hat tip to
@Bob Velke). Just look at the note below -
View attachment 392679
Henry Clay Price born about 1830 in Kentucky, was a farmer in Monroe County, Missouri. In the 1860 Census he's married to Ann (records show Henry married Ann Louisa Hickman in 1850) and has two children, Sterling age 7 and Lavena age 3. He's prosperous with real estate worth $2,000 and personal property worth $3,800. That likely includes the 9 slaves linked to him in the slave schedule. He puts together a cavalry unit in 1862 which he names Captain Henry C. Price's Company. It becomes Company M of the 1st Regiment of Northeast Missouri Cavalry. There's a long string of records for him as Captain.
In 1863 the CSA reorganizes the Missouri cavalry units (I didn't look into this in detail but I bet someone on CWT can give us the details). Price is elected Captain of the new Company G, 3rd Battalion Missouri Cavalry. Again we have records that he's doing this job. The battalion is sent west and - for whatever reason - Price goes off and turns rogue. He had been on a recruitment drive before that - maybe his family needed money? For whatever reason, we get the note above stating that he stopped at the Mississippi and turned to "freebooting" i.e. plundering.
So - what happens next. Well, in 1870 wife Ann and the two children are on the farm in Monroe County. And one Henry Price, age 39, born Kentucky, is in the penitentiary in Jefferson City, Missouri! After that I lose him.
The say the tour of the old penitentiary is lots of fun - you might really enjoy it
@OldSarge79 if you can gloat about the guy who raided your family locked up there!