Lee said the following, and I think it's clear that the bottom line for him was not the Union, but Virginia. The phrase, with slight variations, "Save in defense of my native State, I have no desire ever again to draw my sword" occurs over and over.
Arlington, Virginia, April 20, 1861.
" My Dear Sister: I am grieved at my inability to see you. ... I have been waiting for a 'more convenient season,' which has brought to many before me deep and lasting regret. Now we are in a state of war which will yield to nothing. The whole South is in a state of revolution, into which Virginia, after a long struggle, has been drawn; and though I recognise no necessity for this state of things, and would have forborne and pleaded to the end for redress of grievances, real or supposed, yet in my own person I had to meet the question whether I should take part against my
native State.
"With all my devotion to the Union and the feeling of loyalty and duty of an American citizen, I have not been able to make up my mind to raise my hand against my relatives, my children, my home. I have therefore resigned my commission in the Army, and save in defense of my native State, with the sincere hope that my poor services may never be needed, I hope I may never be called on to draw my sword. I know you will blame me; but you must think as kindly of me as you can, and believe that I have endeavoured to do what I thought right.
"To show you the feeling and struggle it has cost me, I send you a copy of my letter of resignation. I have no time for more. May God guard and protect you and yours, and shower upon you everlasting blessings, is the prayer of your devoted brother, R. E. Lee
---------------
Arlington, Virginia, April 20, 1860.
"My Dear Brother Smith: The question which was the subject of my earnest consultation with you on the 18th inst. has in my own mind been decided. After the most anxious inquiry as to the correct course for me to pursue, I concluded to resign, and sent in my resignation this morning. I wished to wait till the Ordinance of Secession should be acted on by the people of Virginia; but war seems to have commenced, and I am liable at any time to be ordered on duty which I could not conscientiously perform. To save me from such a position, and to prevent the necessity of resigning under orders, I had to act at once, and before I could see you again on the subject, as I had wished. I am now a private citizen, and have no other ambition than to remain at home. Save in defense of my native State, I have no desire ever again to draw my sword. I send you my warmest love.
"Your affectionate brother,
"R.E.Lee."
-----------------
Arlington, Virginia, April 20, 1861.
"General: Since my interview with you on the 18th inst. I have felt that I ought no longer to retain my commission in the Army. I therefore tender my resignation, which I request you will recommend for acceptance. It would have been presented at once but for the struggle it has cost me to separate myself from a service to which I have devoted the best years of my life, and all the ability I possessed.
" During the whole of that time — more than a quarter of a century — I have experienced nothing but kindness
from my superiors and a most cordial friendship from my comrades. To no one, General, have I been as much
indebted as to yourself for uniform kindness and consideration, and it has always been my ardent desire to merit your approbation. I shall carry to the grave the most grateful recollections of your kind consideration, and your name and fame shall always be dear to me.
" Save in the defense of my native State, I never desire again to draw my sword.
" Be pleased to accept my most earnest wishes for the continuance of your happiness and prosperity, and believe me most truly yours,
"(Signed)
"R. E. Lee."
I tend to question whether Lee would have said anything about not fighting the Union, because it would have been obvious to him if he had to "draw his sword", that it would be used against the Union. And he clearly expressed that he would defend Virginia by force. I think Lee did not
want to fight against the Union, but was ready to do so if he felt it was necessary.