It is temporary as lemmy.world was cascading duplicates at us and the only way to keep the site up reliably was to temporarily drop them. We’re in the process of adding more hardware to increase RAM, CPU cores and disk space. Once that new hardware is in place we can try turning on the firehose. Until then, please patient.
ORIGINAL POST:
Starting sometime yesterday afternoon it looks like our instance started blocking lemmy.world: https://lemmy.sdf.org/instances
This is kind of a big deal, because 1/3rd of all active users originate there!
Was this decision intentional? If so, could we get some clarification about it? @SDF@lemmy.sdf.org
It is temporary as lemmy.world was cascading duplicates at us and the only way to keep the site up reliably was to temporarily drop them.
We're in the process of adding more hardware to increase RAM, CPU cores and disk space. Once that new hardware is in place we can try turning on the firehose. Until then, please patient.
Heads up: a new backend issue has been opened to track performance degradation: lemmy#3466. Looks like there's also a private matrix channel that sprouted up for performance tuning & troubleshooting that you might potentially be interested in joining: info.
increased cores and memory, hopefully we never touch swap again
dedicated server for pict-rs with its own RAID
dedicated server for lemmy, postgresql with its own RAID
lemmy-ui and nginx run on both to handle ui requests
Thank you for everyone who stuck around and helped out, it is appreciated. We're working on additional suggested tweaks from the Lemmy community and hope to let lemmy.world try to DoS us again soon. Hopefully we'll do much better this time.
Killer stuff! Sorry for my contributing undue pressure on top of what was probably already a taxing procedure happening in the server room.
Out of curiousity: how do you feel about Lemmy performance so far? I'm actually a little bit surprised that we already managed to outstrip the prior configuration. I suppose that inter-instance ActivityPub traffic just punches really hard regardless of intra-instance activity?
Have a tanoy/horn announce icon at the top like with Mastodon where status information can be posted.
Change the heart icon to link to a method that supports the local instance
Attempts were made to create a thread for the almost daily upgrades we're going through with BE and UI changes, but even with pinning it doesn't have the visibility.
We're on site in about 1 hour to install a new RAID and once that is completed we'll finish the transfer of pict-rs data.
I have created a sub called MetaLemmy in hope that people will use it to discuss matters like these, so that the SDFpubnix sub will be more useful for announcements and such. If you want to discuss the meta-workings of lemmy.sdf.org, please go there (and if you are interested in moderating, please respond to the post there, as I do not want to be the mod of that sub long-term.) Thanks.
If this is true I will be making another account and moving to a different server. The only reason I joined this one was that it doesn't seem to block/defed.
You're absolutely welcome to do that as it is your decision and you have many choices. We hope to build a community of folks that would like to help the fediverse grow and support smaller instances. Similar growing pains were seen during the twitter exodous last September.
As someone working on integrations I'd also have to leave if we're permanently blocking the largest instance, but I'd like to give SDF the benefit of the doubt here. Maybe this was just a temporary measure to deal with the insane load from yesterday?
It's an absolutely massive instance that's a net negative for the Fediverse. It completely defeats the purpose of federation. The Mastodon devs used to drive people to smaller instances but decided they wanted to be /the/ instance. They made themselves a default in the app. At around the same time other instances started getting a crap ton of spam from them that ate up a bunch of moderator's time on smaller instances. The Fediverse only works in regards to moderation because there is less users per admin, but mastodon.social doesn't have that advantage. A bunch of people defederated from them as a result which was a good thing for the instances that did it. They failed pretty hard at communicating during this time as well.
Having one instance hold a large part of the network is bad for everybody involved. Defederating from monoliths is a healthy thing for networks to do. Building your own web beats any algorithm, can't do that if you're already federating with 99% of people.
It's the biggest instance by far, to the point of being virtually non-viable to defederate from. If they ever sell out or make decisions other instances don't like, tough cookies.