It also had to do with views on the construction of the Union. Southern statesmen argued that the ultimate sovereignty wasn't with the Federal government but with a singular state's convention, with the Federal Union's authority derived from a Constitution ratified by said conventions. And that a state convention could legally and morally withdraw their consent to be governed from the Union.
I must highly recommend William W. Freehling's
"The Road to Disunion", a two volume work that examines the cultures and societies of the South (the South was not one monolithic bloc, although over slavery it could seem so), how the slavery institution was written about and ultimately defended by Southern thinkers and politicians, and how slavery itself influenced Southerners (things like how the ever-present tension of master vs. slave, in terms of "Did the slave break that tool on accident, or on purpose? Are they really sick or are they faking?", led many slaveholders to become suspicious individuals toward others as well). The second volume in particular delves deeply into the mechanics of how things developed toward secession and how it happened.
And, of course, the legal and extralegal means employed by the slaveholders and their allies to protect slavery from perceived threat, resulting in violent reaction to even perceived anti-slavery sentiment and poisonous loyalty politics in the South.
The link on Amazon if anyone wants it:
Volume 1 and
Volume 2.