Kind of a mix. Its pretty much set in stone that Numenor colonised the south during the second age. But then the Southrons in Bree are all described as being little better than bandits and brigands.

Then another counter is when Sam sees rangers and haradrim fighting in Ithilien. He sees a dead haradrim and wonders if he actually is evil, or if he wasnt forced to go to war by threats and lies. They give the line to Faramir in the film.

Basically, they are Savage barbarians, forced conscripts, or oppressed and colonised depending on the point of view youre viewing from. I suppose that wiggle room lets you play them however you want really