I have to disagree with the part about the Lee name. It did, in fact, "hold him back" in many ways. The family was impoverished, had to rely on the rest of the family to scratch by, and probably exacerbated his mother's health problems. He was considered unsuitable, remember, by Custis as a match for his daughter. I have no reason to believe he wouldn't have faced the same opposition from other fathers, had he wanted to marry them. What Lee did is to make himself impervious to the scandal and spend his entire life trying to overcome (in his own mind, if not others) what his father and brother (don't forget THAT scandal, which in many ways is worse than Harry's) had done. The Grant name may not have done anything for Ulysses, but it didn't brand him. There is no burden, for someone who comes from a place where one's name is a key in who and what he is--like the burden of one's immediate family. Maybe it's a southern thing, but you are (were) known by who your family is/was.
Lecture over.