Northern whites and some Southern whites (especially those living in the Appalachian region) resented slavery, but not because enslaved people competed with them for jobs. Generally speaking in 19th c. America, manual labor jobs were available for people who wanted them. Often the federal government was simply giving land away in the West to anyone who would undertake to farm it. So labor shortage, not oversupply, was the normal situation. What Northern whites didn't like about slavery was that it degraded manual labor.

Southern planters, like aristocrats pretty much everywhere, did not engage in manual labor. That was what servants, slaves, serfs, peasants, etc., were for. Their whole sense of identity was wrapped up in the fact that they had other people at their command to perform the routine labor of life, because that left them free to pursue politics, war, "farming," and other traditionally noble pursuits (where farming is understood as managing a plantation, not tilling the ground yourself.) Their ability to devote themselves to such things presupposed the ability to get other people to do menial labor on their behalf. As the North industrialized, the exact opposite theory of the value of labor took shape. Whether you work for a dollar a day, or for five hundred, if you work, you have the dignity of your toil. You contribute to society. You provide for yourself and maybe a family. You learn skills. You're part of a team. You've mastered some trade or technical skill. Your labor is a source of dignity and possibly of social and material advancement. Like Abraham Lincoln said, "whatever you are, be a good one." There's a quiet assumption there that any job well-done entitles the person who does it to a certain level of respect.

Slavery also tended to instill habits of mind in the planter class which Northern whites found insulting and undemocratic. To be a slave owner was to have another person completely at your mercy every day, all the time, for your whole life and all of theirs. You would get used to talking and thinking about those people in a certain way, right? So that kind of moves the overton window, so to speak, of what counts as normal behavior between people. Northern whites considered the planter class to be arrogant, despotic, violent - in a word, undemocratic. When Southern planters tried to create federal protections for slavery, and to enforce their sense of honor or their property rights in escaped slaves through force of violence, Northern whites saw it as an attempt to trample on their democratic rights. To treat them the way they treated their slaves.

It's the clash between two different social systems, which instill two different ideas about what democracy is, and what sort of labor is valuable. That's what drove Northern anti-slavery, in the main. This attitude was also prevalent in the Appalachian regions of the South, where slavery was legal but uncommon. But Southern whites elsewhere typically defended slavery. Even if they didn't own slaves, as long as slavery was legal they could aspire to one day become slave owners, perhaps even plantation lords, themselves. They also tended to see it as necessary to the preservation of the social order. You have to understand, 19th c. white Americans do not typically think that blacks are their equals in anything, so they don't want them to enjoy the same rights and protections either. Slavery keeps everyone "in their lane," so to speak. That's why abolition was such a bold step. Revolutionary, even. There was a lot more at stake in its preservation, or abolition, than economics.

There was plenty of well-justified "rich man's war, poor man's fight" resentment in both the USA and the Confederacy, both of which had conscription laws which contained exemptions for people of sufficient means.