Of course the use of scouts goes back at least as far as Moses before he entered what is now known as Israel.
True basically a spy doesn't wear a uniform and will chat up the locals but if caught will leave this veil of tears.
It wasn't unknown for scouts to wear the enemies uniform to elicit information but that was definitely taking a gamble if caught .
Why didn't Grant and Sherman know why they were about to be attacked at Shiloh or Howard who ignored reports of Jackson about to attack XXI Corps? War isn't always so neat and clean stuff happens.
As far as Shiloh, I remember reading that Halleck had actually ordered Grant not to send out scouts or to scout aggressively. At the time even, I thought that it made no sense whatsoever. That combined with Grant's decision to drill his army to the exclusion of building any type of fortifications which would have rendered the surprise attack ASJ launched a worthless attack.
In actual fact one of Sherman's subordinates did send out patrols. When he reported the fact to Sherman he was threatened with court martial and removal from command if he repeated such nonsensical reports to him.
As far as Jackson's famous flank attack, it was not just Howard but the entire AoP leadership which ignored not just scouts, but direct observation by their balloons, who could literally see the long line of troops marching to retreat down the Orange Plank Rd. The problem was not unawareness of the opponent, but such complete overconfidence in the success of your own plan that you ignore the possibility of your opponent making a plausible counter to your moves.