Usually when "XXXX" raised a regiment they applied for the regiment to be accepted by a State into State service and then arms would be drawn from the State's inventory. Before the war a State would apply for arms to be delivered to them from the Federal Government under the militia law from 1792, and the Federal Government would send the arms and if there wasn't a State arsenal, they'd be kept in another spot, some States like Tennessee, (that is if I'm remembering everything right), even kept the arms in the basement of the State Capitol.
It was common for obsolete arms would be issued by a State, and Federal/Confederate Governments would exchange those freshly issued guns State owned arms with newer Federal/Confederate Government guns when that State militia/volunteer regiment was accepted into Army service. In theory those State owned guns would be returned to the respective State, but that didn't always happen. There are many instances of States North and South sending letters to DC or Richmond demanding the guns be returned or replaced so they could arm new regiments for State defense.
In this link there is articles on Southern State militias and volunteers getting ready for war and most go into detail on where and how many arms States had and kept them:
All that being said there was some instances of officers, or soon-to-be ones buying surplus weapons for they're units, mostly up in the Midwest and then getting trying to get reimbursed for it. There's the famous scandal where General Fremont bought the Army's own surplus guns from unscrupulous dealers at extremely marked up prices, while the Army still had the guns in their possession! (It's how a modern bank got its start!

) and Fremont even bought a large amount of surplus Austrian muskets, then had them converted to percussion with the bill being presented to Federal Government afterwards.
I'm no expert and probably misremembering a couple things, but there's a lot of interesting stories related to this question.