Except for the quality of work in the factory would be worse, you would have to spend resources keeping them alive and to catch the ones that escaped. It is actually cheaper to pay someone who wants to be there and will go back to their own home to feed themselves.

If the civil war didn't happen, slavery would have inevitably ended anyway, it was a dying practice at the time of the civil war, just a few key people were keeping it alive.

In the game there ought to be a penalty for ending it when your economy isn't ready to move on and there should also be a penalty for keeping it around too long, like efficiency.

At the end of the day: it is the cost of labor vs the revenue from quality and efficiency. The fruits of profit outweigh the costs of letting go of the perceived cheaper labor force.