But as you also assert, the oath was to the various “United States” then it applies equally to all states in the United States at the time the oath was taken. Further, your claim that Lee did not break his oath rests on Virginia and the other states in rebellion being no longer a part of the United States. Therefore Grant did not violate his oath, as he was attacking a rebellion in the US view and a foreign country in the secessionist view- either way, not breaking the oath. However, the oath Lee took, as you’ve outlined, applies equally to Maryland and Pennsylvania as it does to Virginia therefore any former US officers having taken that oath then fighting for the confederacy who set foot on US (non-seceded) soil did indeed break said oath in both the US and secessionist view of the matter. They took an oath to defend the United States (individually) in your view then attacked some of those states.
In terms of Grant or other Union officers, the are either suppressing an illegal rebellion within the United States, in line with their oath, or attacking a foreign enemy of the United States, also in line with their oath.
Either way you want to frame secession or the legitimacy of the CSA, Lee broke his oath when he crossed the Potomac, Grant did not violate the oath when he either invaded a foreign country who was an enemy of the US or put down a rebellion inside of the US.