If you mean to choose a Yet Another Small Community of Users, there's never been a shortage of those, and there's barely a need for the federated space for that.
If you don't want to see how abusing power to remove and drive users away from communities removes and drives users away from communities, I don't know what to tell you, but I think people are coming to the federated space seeking alternatives to reddit and twitter, not just small communities, so unless something is done about the federated architecture, what happens to access to the biggest communities will always matter in regards to population numbers.
If a user gets their account banned and purged for barely any reason but extreme escalation, I think that will always be a concern - you have no assurance that users will simply create another account and remain. If there's going to be abuse by the leadership in the alternative, why would anyone who comments on RedditMigration simply remain in the fediverse and not go back to Reddit? If you are going to get falsely accused accused and banned for shit reasons, even to the point of an admin of the most popular instance fabricating accusations that you are an alt of a CSAM account, why would most people remain?
Nothing is stopping the user from accessing lemmy.world communities from the alternative instance they chose (unless it has defederated). It's just that in this case lemmy.world did not want to be responsible for the user's content.
Abusing admins is nothing new and with reddit you have absolutely 0 recourse besides making a new account at the mercy of the very same admins. On Lemmy at least you can select another instance and still access the same content.
My point is, if you are going to be dealing with it anyway, why would you participate in the social network that is order of magnitudes smaller? You can access content on reddit without an account, the problem is participating alongside the community.
For all intents and purposes, since you are still locked out on lemmy from doing because of its server-centric communities regardless of which instance you choose, it boils down to the same outcome for the same desired goal - creating an alt - except with an order of magnitudes smaller reward - far less population and engagement than on reddit.
So rather than sticking to lemmy, it seems natural that people go back to their old but bigger platforms.
Federated is great for maintaining persistence of your account beyond the whims of fickle admins, but that's a tertiary problem. No one is that exited about keeping their user history, they are excited about participating with everyone else about the topic that is being discussed.
It's not worthless, but I can understand it explaining some of the decreasing population numbers if they encounter it even just once after months of participating on the platform because of how disruptive it is. No one is normally going to stick around a community when only the fraction of the local users in your own lemmy instance can view it.