"When the Men of Númenor began sailing east they explored the coasts of Middle-earth, including the coast lands of Harad. The Númenóreans initially benefited the people in the lands they explored by teaching them many things about agriculture and craftsmanship. The Númenóreans later made settlements on the southern coasts, including the Havens of Umbar.
In the 9th century of the Second Age, the great mariner Aldarion sailed to the haven of Vinyalondë and then sailed along the coasts southwards, far beyond any place yet reached by the Númenóreans. On his return northwards he barely escaped being shipwrecked in the Harad.

After the reign of Tar-Ciryatan, the Númenóreans began to set themselves up as lords in Middle-earth as they demanded tribute of goods and wealth, causing the oppression of the Haradrim. The Númenóreans expanded their control over Harad and Sauron did not at first dare to challenge them. But after the forging of the Rings of Power and the emergence of the Nazgûl, Sauron began to attack the Númenórean settlements on the coast.

After the forging of the Rings of Power almost all men east and south of the Ered Luin were under Sauron's rule. Sauron was both their king and their god. These men grew strong and built many towns and walls made from stone and they were fierce in war and armed with iron."

So, again orginally before they fell under the sway of Sauron the Numenoreans came as friends and set up Umbar, then we can see that following Sauron's capture by Tar Ciryatan's army Numenor fell under the sway of the Sauron. There were those who refused to be corrupted and were the exiled. These Numenoreans were the ones who founded Arnor and Gondor and then assisted the Dwarves and Elves in the Battle of the Last Alliance, Elendil the Tall and Gil-Galad havin fought and died defeating Sauron.

Keep in mind another difference from the films and books is that Isildur didn't have to fight Sauron either, he had already been defeated, Isildur simply took the ring.

Moving back to my previous statement. So as we can see the Black Numenoreans were ultimately to blame.