The state right to leave the Union was the core of the dispute of the Civil War.
The Southern US, at that time, had a number of politicians who argued that they had that power. Unionists, although their population was probably as numerous as secessionists given the popularity of the Constitutional Union party, were either removed from office or supported the local governments which seceded. The North was also mixed, with both Unionists and those that took a range of other approaches, for example not believing the federal government had a right to invade a US state.
Why did these states want to leave? Well it wasn't for the right to have slaves. They already had that right. It was because of the election of Abraham Lincoln, as well as the decades long political conflicts between free and slave states in places such as Kansas, Harper's ferry, and across the north with the fugitive slave act.