It's not just that. When the US was formed, many people thought it was a coalition of individual nations, not one, individual nation. The South felt like the federal government was not representing them (thanks 3/5s compromise), and so they wanted to secede. So, when people were choosing where to fight, some people felt more allegiance to their state than to the federation, even if they disagreed with slavery. Robert E. Lee is a good example. This is also why he worked to reunited the states after the Civil War. It definitively answered the question: The US is one nation. None of that takes away from the fact that slavery was the primary issue that was dividing the country.

---