YSK: lemmyverse.net is the best way to find active communities
YSK: lemmyverse.net is the best way to find active communities
You should know this because finding communities on lemmy can be tough.
You're viewing a single thread.
I dont get why a third party site can see more of the fediverse than the fediverse can.
A brand new instance on the fediverse doesn't initially know about any other instances. Only when someone searches for a community@instance does it then go talk to that instance and subscribe to get posts/comments etc.
Maybe you can help me. I joined under Lemm.ee. I want to subscribe to c/catswithjobs@lemmy.world but it doesn't show up on my search list. What do I type in to find it?
Search !catswithjobs@lemmy.world. Make sure to include the exclamation mark. Give it about 30 seconds and search it again, and the community should appear, and now you can subscribe.
Thanx for the reply! For some reason, I didn't get the results in my search bar via the Connect app, but I was able to subscribe via my desktop. Not sure what the difference is, but at least I can see my working cats now :)
I'm also not sure what the difference is, but the desktop app does seem to be better about initial federation. Glad you were able to get your working cats, feel free to ask if you have any other questions
It sees the same information, just aggregates it differently I guess.
They explain it on the project's GitHub:
How does discovery work?
It uses a seed list of communities and scans the equivalent of the /instances federation lists, and then creates jobs to scan each of those servers.
How long till my instance shows up?
How long it takes to discover a new instance can vary depending on if you post content that's picked up by one of these servers.
Since the crawler looks at lists of federated instances, we can't discover instances that aren't on those lists.
Additionally, the lists are cached for 24 hours, so it can take up to 24 hours for an instance to show up after it's been discovered till it shows up.
And why is it using this method, and not the third party sites method that sees more?
The Fediverse (Lemmy/Mastodon/etc) is based on a following/subscribing model; each instance only "sees" what it's users are currently following or subscribed to. This keeps storage and systems usage lower since each instance doesn't need a complete copy of the entire Fediverse. This third party is more like a web crawler like Google, just crawling from instance to instance and saving the data. Hopefully in the future Lemmy could add something like this discovery feature, maybe something like Mastodon Relays, to aggregate community lists, but it would definitely put more strain on each instance.
Same here. I understand the whole notion of how an instance is able to "see" communities on another instance on the fediverse. But I don't get what these kinds of website do differently to see all communities on all instances and why instances can't do that directly.
What they are doing is pretty much what Google is doing to find the webpages. It's called crawling.
Appreciate the clarification! I've been trying to wrap my head around how the fediverse works but I'm far from a programmer so some parts are still a bit confusing to me.
Is it just a different approach to reach the same goal or is there some inherent limitation to how the fediverse works that prevents instances from using crawlers?
Lemmy a little bit comparable to E-Mail. Both are decentralized systems. I can register a domain ajsldkfjkadsfjkladfsjklfasdjöfda.com and host an email server there. Now, I go to gmail and send a mail to hi@ajsldkfjkadsfjkladfsjklfasdjöfda.com. Gmail has had no idea that ajsldkfjkadsfjkladfsjklfasdjöfda.com even existed, but gmail and ajsldkfjkadsfjkladfsjklfasdjöfda.com speak the same protocol and can therefore communicate. So, the email reaches me. You see, Gmail did not have to crawl the whole internet to find my new site. It being able to speak the same language (some protocols) is already enough.
Same with lemmy: Different instances (e.g. lemmy.world and feddit.de) don't need to know which instances exist. If I (from feddit.de) want to post on lemmy.world, it will work, because they are having the same protocols.
What prevents instances from using crawlers?
Nothing. It's hypothetically possible, but:
- It has to be implemented in a good way
- In a large network, it would take some time to crawl.
- Where will the crawling be done?
- On your PC or phone? Then, the crawling will be done for each user. But maybe, this takes too long.
- So, let's do it on the lemmy instances...? What if an instances blocks the IP Address of another instance? Then, the users wouldn't know about the communities there..
Maybe, there is already some crawling happening. The search of the lemmy app WEFWEF is able to find communities of other instances. Either, WEFWEF shows me posts and stores the community they come from somewhere and seaches them when I use the search... Or it accesses a crawled database or similar to suggest me those communities...
Wow, thank you so much for taking the time to write such a detailed explanation. It really clears up a lot of what I was confused about.
The community across the board on Lemmy has been so refreshing compared to the last few years on Reddit or any of the alternatives I'd tried before.