You don't have to reconcile it, because Lee did not go to war for slavery. He left his reasons for us to read.
Still, a Union that can only be maintained by swords and bayonets, and in which strife and civil war are to take the place of brotherly love and kindness, has no charm for me. I shall mourn for my country and for the welfare and progress of mankind. If the Union is dissolved, and the Government disrupted, I shall return to my native State and share the miseries of my people, and save in defence will draw my sword on none." - letter to his son Custis, Jan. 1861
God alone can save us from our folly, selfishness and short sightedness. The last accounts seem to show that we have barely escaped anarchy to be plunged into civil war. What will be the result I cannot conjecture. I only see that a fearful calamity is upon us, and fear that the country will have to pass through for its sins a fiery ordeal. I am unable to realize that our people will destroy a government inaugerated by the blood and wisdom of our patriot fathers, that has given us peace and prosperity at home, power and security abroad, and under which we have acquired a colossal strength unequalled in the history of mankind. I wish to live under no other government, and there is no sacrifice I am not ready to make for the preservation of the Union save that of honour. If a disruption takes place, I shall go back in sorrow to my people and share the misery of my native state, and save in her defence there will be one soldier less in the world than now. I wish for no other flag than the 'Star spangled banner' and no other air than 'Hail Columbia.' I still hope that the wisdom and patriotism of the nation will yet save it. - letter to Markie Williams, Jan. 1861