Is it just pure racism? Is it just overall hatred of poor people? Some combo of the two?
Combo of the two. There's plans to build condos near my home and my dad flew off the handle at the news because he thought it was affordable housing when it wasn't. No kidding he literally said along the lines of "I didn't move out of the city to have n-words come live near me". He's a racist piece of shit who's lizard brain is on that white flight.
White suburbanites expect their home's value to increase indefinately. It's an "investment." Even though it doesn't fit out conventional understanding of capital, it functions similarly. They come to believe it's their right to derive money from their ownership of property, and will fight as hard as they can to ensure both the value and their ownership of the property.
When social life is organised for the production of commodities people will start identifying with their their commodities. Once they tie their personal value to that of their commodity, such as a house, an assault on the value of their property is seen as equal to an assault on their person.
Housing becomes more than a commodity under capitalism. It's your retreat from the workplace, from exploitation, from capitalism. It's a coping mechanism based on intentional ignorance. Having poor people, living embodyment's of Capitalism's failures, shatters that reality.