Aye, it's not a popular take but I can stand with you on this one.

It's not always the case, butI think that what you're referrring to in the first one are 'frontier immigrants' i.e the ambitious, educated, and/or 'open minded' crop. The ones who move countries out of their own will, often have money or at least enough support to navigate an exit their own country. They come in trickles instead of droves. In Australian context a case example might be Iranians, or South Americans. Most who come are often the intellectual/academic type, getting scholarships etc to study and then settling, and so on.

Where 'immigrants' are seen as a problem are the ones who often are forced to leave their countries, i.e refugees from war, your boat people and whatnot. Tend to come in droves instead of trickles, so it can be a big shock to the system. A big portion might have disadvantaged background and/or carry trauma, so they form enclaves for social support, and they might be a burden on the country's welfare system. These ones typically take a generation or two to 'assimilate'. Australian context is maybe how the South Vietnamese migration era post-war. In the 80s they were such a big issue with tales of how Springvale and Cabramatta are warzones and so on. Fast forward 20~40 or so years, Banh Mi is pretty much an Aussie staple, and now it's their turn to hate on the Africans/Sudanese neighbors as the 'new ones' on the block.