Where Ellsworth was shot down, was at the Marshall House, a boarding house in Alexandria City, Virginia -- Marshall ran up a Confederate National Flag and it was seen from the White House unobstructed. Being that the location is so close to the Washington Arsenal (Fort Leslie P. McNair now days), it was deemed as a threat.
Ellsworth's men were security detailed at the White House and Lincoln and his sons were extremely fond of this Colonel and the renowned bravery of Fire Fighters who joined him from New York (aka Fire Zouaves) were part of a detail by then Colonel Stone, in charge of the D.C. Militia charged to secure the Long Bridge (where the railroad bridge is now located and near 14th Street Bridge) and secure Alexandria City. The U.S.S. Pawnee was also detailed to this movement, as canon shot could reach the Arsenal from Alexandria's shores. (Again, it has to be reminded that nobody had visions of a civil war or disunion).
Ellsworth having climbed the steps to bring the CSA National Flag down off the flag pole, was met by Marshall and shot, and the Sergeant with Ellsworth shot Marshall in return but, the death of Ellsworth, in my opinion did more to burn a larger 'anger' in Lincoln had the incident not take place.
Further, post war stories about 'then' Colonel R. E. Lee voicing clearly his desire that no secession take place and influential as he was, I am of the opinion had the Marshall-Ellsworth incident not taken place, which was before First Bull Run/Manassas, history's path might have been altered. There was little resistance by the Dominion Rifles that were the militia unit for Alexandria, City -- Their commanding officer would be suspect and nearly captured and executed had it not been for General Lee's personally stepping in - this man, Montgomery Coarse would later become a General in addition to being Lee's banker along with his brother who remained in a civilian capacity and ran the bank when Alexandria was occupied and placed under Marshal Law until the close of the war.
What I find interesting though, few remember or recognize Alexandria City as being the first captured, first occupied city in Virginia and before First Bull Run. Colonel W. T. Sherman's "West Point Artillery" was assigned on the banks of the Potomac on the Alexandria, Virginia side prior to being part of the march to First Bull Run and placed in charge of Infantry.
Beauregard, to whom caused the withdraw of General Johnson's Army, which included Longstreet, JEB Stuart and other future important Generals, to include T. J. Jackson (Stonewall), etc., snatched the only opportunity for Longstreet and Stuart to recapture Alexandria and possibly sack Washington after the route the Union Army from the battlefield of First Bull Run. Longstreet's headquarters was in Falls Church and Stuart had his artillery positioned close to shell Bailey's Crossroads. (Bailey's Crossroads was the winter home of Bailey's Circus which would merge to become Barnum & Bailey's Circus). It is where I can appreciate the effort of General Longstreet in relieving Alexandria but, ordered from superiors away. General Longstreet would soon after recommend Colonel JEB Stuart for the rank of Brigadier-General through General Johnson and would be approved. It is why I have a special place for Longstreet and sad that he wasn't given good credit for all the efforts in liberating Alexandria City.
However, history is what it is and not 'what ifs.' I can appreciate the need for Lincoln to have a buffer around Washington as the banks on the Virginia side are higher and could lob artillery into the city and destroy the functions of Government, et.al. The working City wasn't all that big. Farms were beyond "H" Street N.W., and thus rather small compared to what most are familiar with today.
Just some thoughts.
M. E. Wolf