Force is equal to mass times the velocity squared. The six pound shot would have been going something just under the speed of sound when it struck the target. Brick or stone is brittle. It was possible to bang away at a masonry wall & make a breach. There are many examples of that happening during the age of blackpowder warfare.
Dirt is an excellent shock absorber. Which is why forts built in the artillery age have large earthen berms.
View attachment 364541
The earthen berms of Fortress Rosecrans in Murfreesboro TN was built to this design.
The glacis, exterior slope I superior slope are designed to deflect a cannon ball. The mass & shape of the berms could absorb a large number of impacts by low trajectory six pounder cannon fire. The only vulnerable area of the design is the very top edge of the interior slope. That is where a head log would have been installed. The circular error of a smoothbore six pounder at 800 yards is about the size of a barn door. Even at closer ranges, an 1841 model six pounder which was ubiquitous in the frist years of the war, was not accurate enough to strike the headlong with regularity.
It was Mr. Newton's laws that gave the 12 pound gun-howitzer Napoleon its wallop. Once again, banging away at earthen fortifications would be a largely futile exercise. Smoothbore siege guns are huge in order to bring the second law of motion into effect. Rifled guns could strike a target with exponentially greater pounds per square inch of force. They were also accurate. At the 800 yard range, a rifled gun could hit the window in the six pounder's barn door over & over again.