If immigration leads to more unemployment, then that is an economic problem, especially in the hypothetical case where the social benefits system is getting more and more strained by an influx of unemployed people. But generally, I think that you can expect that the immigrants will soon find employment. Besides that, there's the cultural aspect that @jet@hackertalks.com mentioned. You could also make the point that the country's infrastructure is more and more stressed as the population grows, but that is fixable and potentially counteracted by the labour potential of the immigrants themselves (i.e., qualified immigrant work forces can make a large-scale infrastructure overhaul possible that will lead to greater national capacities and a net benefit for the entire population).
Aside from these things, I would argue that most of the other reasons boil down to xenophobia or racism.
but that is fixable and potentially counteracted by the labour potential of the immigrants themselves
That's how I would deal with immigration in my power fantasies. I'm sure in reality it's much more complicated than that, but the basic idea of bringing in immigrants and using their labor to build more infrastructure (and paying them a fair wage for it) seems sound. Coupled with pro-housing policies and free education - not necessarily college but trade school and language classes.