According to this letter written in 1863 "The camp is about four miles from the Long Bridge, and is situated on Arlington Heights
not very far from the late residence of the far-famed rebel Gen. Lee."
The letter is at the Civil War Washington website -