Is investing in real estate immoral if you use it to buy your first home?

I'm someone who believes landlording (and investing in property outside of just the one you live in) is immoral, because it makes it harder for other people to afford a home, and takes what should be a human right, and turns it into an investment.

At the same time, It's highly unlikely that I'll ever be able to own a home without investing my money.

And just investing in stocks means I won't have a diversified portfolio that could resist a financial crash as much as real estate can.

If I were to invest fractionally in real estate, say, through REITs, would it not be as immoral as landlording if I were to later sell all my shares of the REIT in order to buy my own home?

I personally think investing in general is usually immoral to some degree, since it relies on the exploitation of other's labour, but at the same time, it feels more like I'm buying back my own lost labour value, rather than solely exploiting others.

I'm curious how any of you might see this as it applies to real estate, so feel free to discuss :)