People have a wide variety of opinions and interests that are not represented well on a two dimensional left right spectrum.
Lemmy has a fair few people who will fight you over their own perceived interests.
Because Lemmy is so small it's common for the same people to argue over their interests in every community. In the morning I catch up on the all feed in 20ish minutes (I block politics and memes).
Take that together if you say something against any groups zeitgeist or one of their trigger topics it feels like all of Lemmy is dog piling you.
Remember online isn't the same as real life, express yourself and share your earnest ideas and don't care about downvotes. If someone is being unreasonable just ignore/block them.
Honestly, avoiding the all feed may be better for my enjoyment of the platform. On Reddit I used to just sub to my specific interests and never interacted or saw all the politics talk or general memes. When I moved here, most communities were too quiet for it to take up the same spot for me, so I started browsing by all instead.
I'll give it a shot, thank you for the reminder/realization.