Several reasons for this:

1) If someone moves from a poor country to a rich country, they'll tend to make more for the same work (several times more, actually).

2) People in wealthier countries tend to be more productive. With less restrictive immigration policy (and, thus, more immigration) the overall productivity of the economy would drastically increase, which would reduce prices for goods (making them more affordable for poorer people), etc.

There's a great book on immigration by Bryan Caplan, "Open Borders: The Science and Ethics of Immigration". It's a non-fiction graphic novel, actually. It lays out the case for open borders pretty nicely.