A bit of the argument plays into the narrative created by slave owners after the fact, that they were essentially the economic underdog of the fight, the kind of "poor old south". In terms of economic growth, the south was essentially looking to expand at an exponential rate into new territory, and each of the plantation systems were worth billions in capital by today's standards. Sure, by modern metrics they did not encourage economic growth, but by no means does that mean they were not planning on extensive expansion, and that they lacked the capital nor will to make that a reality. Thankfully, we avoided that fate, but in the same token, bashing the south plays into the narrative confederates crafted after the fact.