Very interesting that you should bring up Ellis's book--I am reading it right now and I just finished reading The Dreaded Thirteenth Tennesse Union Cavalry: Marauding Mountain Men by Melanie Storie. I have nine relatives who were in the 13th, but they were all originally from Ashe County, NC. I was reading Ellis's book to discover what it was like to travel through the mountains to get to Union lines.
Ellis, I suspect, was looking more for book sales than honesty and to justify his own actions in the war. People on both sides wrote highly prejudicial memoirs to appeal to an audiences that all wanted to hear how awful and depraved the other side was. If you have ever seen Dr. Gary W. Gallagher's lecture series The American Civil War put out by The Teaching Company, he mentions this phenomenon, particularly with regards to Civil War prison memoirs.
Many of my Ashe County relatives moved to Johnson or Sullivan County, TN after the war. I also have Ashe County relatives who were in the 5th NC Battalion-Cavalry that you mentioned--The Graybeals of Ashe Co. had members in both that regiment and the 13th TN Union Cavalry.
I do have 5 relatives who served on BOTH sides: 2 were galvanized Yankees, the other three deserted the Confederate Army, then went to Union lines and enlisted in the Union Army. 4 of these men were from Ashe County.
I must admit I was very surprised when I started researching the Southern part of my family: I had assumed most would be Confederates, but about 50% of my relatives from NC fought on the Union side and all but 1 of my KY relatives fought on the Union side. They were all from Appalachia, which I now have found out did have quite a bit of Union support.
A fun bit of family legend (I can't verify this): Phoebe Graybeal Eastridge of Ashe Co., NC did not want her son Andrew to be drafted by the Confederates. As soon as he turned 18, she dressed him in a hoop skirt to evade Confederate authorities and sent him to Tennessee, where he enlisted in the 13th TN Cavalry.
Next on my reading list is the official history of the 13th by Angel and Scott.