Lee and his wife never seemed able to get in sync. Freeman's biography makes that clear, without unfairly criticizing either of them. Lee's letter, written while on the survey mission in Michigan with Talcott, is revealing: "why do you urge my immediate return, & tempt one in the strongest manner[?] ... I rather require to be strengthened & encouraged to the full performance of what I am called on to execute." There is an added poignancy to reading those words because Lee was, as Freeman points out, completely infatuated by Talcott's wife with whom he had "lifelong Platonic relationship."
I don't see any difference, other than Grant leaving the army for alcohol abuse, and being able to spend time with his wife that way. Before that, he was in Lee's position, but much less happier.
As to Lee's feelings with other women, he was quite playful about in his letter to that Savannah woman he loved so dearly, I forget her name. He teased her for not coming to the wedding haha.
Lee had it all. Good for Grant and his marriage, but there is nothing to knock Lee about in that regard. He took great good care of his wife when she became invalid.