Huzzah! License to ramble!
So... you've seen the story thus far.
Enter at this point a number of Navy officers who were both reform-minded and had enough rank and seniority to be influential; and also their political allies in Congress. (Since nearly all midshipmen were made by Congress, they had a Congressional connection from the beginning, and so long as that politician stayed in office, the officers had that to pull on at least.) The key individuals in the efforts that would eventually be successful (there were a number of false starts and dead ends) were Captain Franklin Buchanan of Maryland, Commander Samuel Francis Du Pont of Delaware, and Senator Stephen R. Mallory of Florida.
Buchanan had been named the first commandant of the Naval School at Annapolis that quickly developed into the U.S. Naval Academy under his leadership. The founding of the Academy had significant Congressional opposition, and that's a story in and of itself... suffice to say that the Navy stole a march on Congress and set up the Academy, and Congress acquiesced. One effect here was to potentially regulate the intake of would-be officers into the Navy, because they had to go through the Academy curriculum first, and be subject to weeding out if they didn't make the grade.
Repeated efforts to create ranks higher than captain failed, until at last, just on the eve of the Civil War, the position of Flag Officer was created. (You can see the lengths to which they went to avoid the word "admiral"...)
The other, crucial wall that needed to be breached was the resistance against a retirement system. Du Pont and Mallory and others came up with the idea of a "Reserve" list that would shunt inactive officers off to the side, out of the main line of promotion, yet not stick them with the stigma of being "retired"-- as Du Pont said, they were still subject to recall at need to take a round for their country. They found an ally in Navy Secretary James C. Dobbin of North Carolina (interestingly, many of the leaders in the reform efforts were of Southern extraction, though not all). Dobbin was wary of causing too much of a ruckus and had to be propped up a bit by the others, but he was at least amenable to doing something, which had not been true of former Navy Secretaries. The result was the Naval Efficiency Board of 1855, which attempted to sort officers into "Active" and "Reserve" categories. Unfortunately, in the process, it was suggested that the board simultaneously recommend officers that should be put out of the service entirely, which mostly undid what the whole notion of "reserve" was trying to do, and a political firestorm erupted. I've told that story elsewhere... [
https://civilwartalk.com/threads/the-naval-efficiency-board-of-1855.85618/#post-662404 , with a bit of repetition of the above ]
When the war came along, and many Southern officers left to "go South," the population pressure on the U.S. Navy officer corps was eased... and then, the opposite condition resulted from the sudden expansion of the Navy, so that instead of too many officers there were too few, and Lieutenants ended up commanding ships again, but for a different reason. (Lincoln's Navy Secretary Gideon Welles did use the pressures of the war to make many of the reforms that had been impossible in peacetime, and the Navy emerged from the war with a full slate of Admirals, a retirement system, and a solid core of officers trained by the Academy.)