I agree. This is my experience with a .58 caliber rifle.
I bought a .58 caliber Zouave rifle off of a co-worker years ago and bought all his ammo and powder and primers. I tried a patched ball and it was extremely tight, especially after some fowling had built up. I almost had to hammer it down the barrel. The Minie' ball was much easier but I tend to believe it was too loose.
The previous owner had installed a vernier peep sight mounted further back on the wood grip. I used the production sights but never worked up a load to get a good grouping. So I just fired it to see what damage it could do. I never even really tried to get the peep sight to work.
I sold it less than a year later as it was just a waste of too much lead and too much powder with litter return on the fun of shooting it. I recently had the opportunity to purchase another one at an estate sale but I didn't even think about it.
Instead, I bought this Dixie Gun Works .50 caliber long rifle. It isn't pretty---which makes me think it was a kit build --- but it is fun to shoot.
View attachment 313014