Lee's loyalty was to Virginia. For most of his life, that meant being a patriotic citizen and soldier of the United States.I found it always hard to explain why Lee swapped sides suddenly - after having expressed some doubts about the secession movement. More than that also his critical stance toward the Union (he had long felt deeply attached to) quite surprised me.
I am currently following just the same idea like you.
I think the loss of Arlington (and it’s turning into a cemetery which made any hope of regaining it illusory)
and especially it‘s impact on Mary Custis had quite an influence on how Lee saw the North - and (consequentially) the Confederacy.
I'm reminded of Decatur's famous toast: "May she always be in the right, but our country, right or wrong." Virginia was Lee's "country" even when she did something like secession that he disapproved of.