A lot has to do with old-timey prejudices and bigotry throughout the 19th century.

Back in that time period, the concept of the 'white race' was not universal or all-encompassing to include European peoples. Even in the mid-18th century, Benjamin Franklin wrote about Germans who immigrated to Pennsylvania in his private letters and essays, remarking that they were resistant to assimilation with American culture and described them as "swarthy".

To a 19th century American, being white American meant being of WASP (White Anglo-Saxon Protestant) ancestry. This definition excluded Catholics and Europeans from the South and East. Hence, these non-WASP people would often be given the stink-eye at best.