The "changes in weapon technology" particularly in the 1870s-80s in the US Army has little to do with the "loose formation", evolution into "fire teams" etc. and far more to do with the response to the very poor performance of the individual infantryman in the Civil War. Again, see what I have posted here on British Army directly before and during the Civil War (You of course have not had the time yet to respond, so not to put any pressure)
British are not alone using close-order against the Zulus, and rather than some oddity of how it survives it makes perfect sense, and why they use square in the mid-1880s; individual or extended order even if they had assault rifles probably isn't the best formation against an enemy that outnumbers them nearly 14 to 1 at Isandlwana; 25,000 total Zulus, discounting those sent to Rorke's Drift vs. under 2,000 British troops almost fully encircled. You are forced to do nothing but get into close order in any era if you do not have heavy guns, did not really fortify your camp out of sheer underestimation and are outnumbered by a ridiculous multiplication factor. Open order was indeed what was mainly used at Isandlwana and did not work, because why would it? At Rorke's Drift, it is almost a 30:1 ratio; extended order even if we were looking at Lee-Metfords being used, I would dare to say, would have been death.
You fail to understand that open order and close order throughout the late 19th century, be it Crimea or against the Zulus, exist simultaneously because infantry has become a combination of prior forms of infantry; that is why the US Army in the Civil War wears the hunter's horn as their literal universal infantry symbol, worldwide known as the old, traditional symbol of riflemen and light infantry, and symbolic itself in its intentional choice for use in this way, why their fatigue uniforms are based at least partially on French chasseurs, why the dress uniform, complete with Hardee has strong German and Austrian Jäger influence, and why the whole army on the smaller scale is using a book literally titled Rifle and Light Infantry tactics, directly translated by William Hardee from a mid-1840s book intended for use with an early pillar-breech rifle-musket, the Model 1846 à Tige, itself preceded by a series of pillar-breech models adopted since 1837; the French are very pioneering in the 1830s & 40s. Literally conducting what the best-drilled ACW Regular US infantry are doing in 1861-65 nearly 20 years later while the Mexican War, mainly fought with flintlocks on both sides is happening at the same time. (Tangent aside, every one of these features I named are very intentional and symbolic in their own right. Even this archaic and outdated US Army of the Civil War we are talking about here, literally models itself among all infantrymen as light infantry, and in practice, ideally fights very flexibly as line or light in the field.)
The perfect antidote to the more well known Zulu War in this regard is the New Zealand Wars that rage during every year of the 1860s, fought wholly without the redcoat, and characterized by extensive trench warfare, bush-fighting that is ideal for extended and open order, fighting against bomb-proofs and rifle-pits; far more relevant given promixity to the Civil War, literally taking place during every year of the war and before and after, and even technologically: See, any source, newspaper, from eyewitness or photographs will confirm directly that the Mauri Wars were mainly fighting with double-barrel shotguns - giving them an extra shot at charging distance, point-blank range - and old British military rifles - perhaps surprisingly, almost all fairly recent, i.e. old P51s or M1839/42 Brown Bess percussion muskets. Double-barrel percussion shotguns seem to be their preferred weapons, and the best-equipped Mauri warriors wore up to three British Army cartridge pouches around their necks for easy access. There are sketches and illustrations (and photographs) of all of this online. The British fought at least half of the conflict with the P53 Enfield, and is a much more symmetrical conflict than the war against the Zulus - the Mauris had their own close-quarter weapons, all manner of short swords and clubs, i.e. patu onewa, but these are used as secondary, close-quarter last resorts; not like the Zulus predominantly using spears and shields with a good throw as their primary projectiles. It was a shooting war, the Mauri occasionally sporting artillery, i.e. during the Waikato Campaign, Ōrākau (most infamous battle). The Mauri had what are considered among some the finest non-European military engineers that had actually learned considerably from the British in wars of the 1840s, i.e., Tītokowaru, developments of Te Ruki Kawiti during the 1840s wars, Wiremu Tāmihana, Rewi Maniapoto, Riwha Tītokowaru and various others during the 1860s wars all come to mind.
The Mauris wore Western clothing for the most part as far as real photographs confirm, and again, technology-wise... while the British fighting against the Zulus have reverted to the RML, with the RML 9lb-er 8 (& 6) cwt guns, some 13lb-ers, the New Zealand Wars of the 1860s are fought with the Armstrong RBLs, some guns up to 110lb (i.e. see the bombardment of Gate Pā).
So if we really want to fascinate over European performance in colonial wars on the scale of infantry, the Mauri Wars are easily much more symmetrical than the Zulu Wars: Typical combat of the Zulu Wars is in close-order, recoats, fought against an enemy who typically charges across open ground protected by a spear that fights with knives and crude spears. British are armed with the Martini-Henry and RML field guns.
A typical Mauri Wars battle (conflict mainly fought from 1860-68) sees British RBL, very occasionally up to 7", bombarding elaborate earthworks complete with underground dugouts with hundreds of shells - sometimes with artillery on their opposing side - and fought in dark blue uniform and fatigue (Kilmarnock at this time) caps, infantry fighting in extended order, and perhaps most shockingly even regularly being bloodily repulsed and defeated by said Mauris. The Mauris are not quite the Boers of the 1st or 2nd Boer Wars, but they are certainly closer to that than even in the middle between the Boers and the Zulus. It is one of the most interestingly symmetrical colonial conflicts of the late 19th century, definitely that of the British. A British soldier in a typical battle in 1879 can expect if he misses, to at least be able to engage a Zulu, armed with spear in melee; a British soldier in a typical battle of 1863 or '64, even if he had a Martini-Henry is not saved by the fact he is in an army of crack-marksmen if he misses his shot in close quarters, as he is ambushed by hidden Mauris hiding in tunnels and dugouts in the trenches he has stormed, and even if a Mauri were then to miss a shot too, he still likely has another barrel he can quickly use to dispatch the British soldier.
Fantastical, imaginary descriptions aside - If any of this comes as any shock there is a lot out there for you to read and research. The point is that the American armies of the Civil War are really not ahead of European contemporary armies. More effort should be put in by those of us who take such interests in the Civil War, if we really are to understand its place in history, and relative to its own time, for I very often see something that sounds like what you express here, some sort of conception of European powers as though they are all using the kind of manuals the Americans took note of in the early 1830s through mid-1840s. Here is a challenge: Find me one military in Europe that is neither Balkan nor Iberian that is using a manual older than what Hardee's is based from (1846). You won't find any because they do not exist. It pays in personal knowledge to really research in detail, not merely to ponder from a distance, the way these other armies fought if we are to make sweeping claims about them relative to the Civil War, or further base our understandings of the Civil War on what we know of their contemporaries. I realize this is an old comment; plenty may have changed for all I know since then, you may know all this and more. If not, there's a lot to start with to research here. I personally do not have anything much to say about any armies I do not even have the slightest knowledge of the infantry manuals of, and without coming across as mean, it would probably be wise to do similar; none of the statements from you I have replied to seem to imply the reading so much as a word of any British manual contemporary to anything Crimean War nor Zulu War era.
On the Mauri Wars, which was not an intended focus of this post, but now seems to be - a far better comparison, objectively, if we really want to have a go at analyzing British Army and generally European tactics, literally contemporary to the Civil War in its entirety as oppose to the Zulu Wars over 13-14 years in the future.
Much of what I am reading here sounds a lot like what I myself thought about 4 or 5 years ago anyways, so who knows. I hope I have managed to share something here worth your time, and ideally this could completely change your understanding - it should, most people know nothing about how any European Army actually fought in the 1860s (or 1850s, or 1870s really..) from most of what I have gathered, whether even certain scholars or basic, mainstream discussion when it comes to Civil War; Civil War buffs are really in a bubble, outside doctrines very obscured. That is likely where the whole narrative of such common myths as "The ACW was fought with outdated tactics, so it was up to the combat of the Civil War to figure it all out the hard way/the Europeans should have learned from the Americans" and others to similar effect which are made - honestly, like much of the first page or two of this thread - in sheer ignorance of what was actually happening, actually contemporary to the Civil War. This post, and some of the other long ones I have posted on this thread are not meant to be much informative so much as to spark interest into an entire ocean for research and knowledge waiting below your feet.
I hope you may, if I have come across the right way, sparked similarly great interest in these things, in you, the kind of sparks of interest that fuel most of our being here to begin with in all things Civil War.