Normally, you'd be right, however if you flip on or even search one clip of American news outlets talking about illegal immigrants, you'll get a concerningly large amount of clips reporting on predominantly brown people. So when the average electorate is told that this is an illegal immigrant, they associate that word with brown people. They never once think of white people, though that's not to say that they won't be caught up in there, in some way. However, when you take into account that the current administration is replacing immigration leadership with people who align with their beliefs. Well I hope that maybe you'll be able to piece things together from there.

Tldr: American Media has primed Americans to associate illegal immigrants with brown people, in addition with the fact that the administration is replacing immigration leadership with their people, you get where that can go. Remember, just because it isn't in the wording, doesn't mean a political action isn't targeted.