7mo ago

Google Is the Only Search Engine That Works on Reddit Now Thanks to AI Deal

www.404media.co Google Is the Only Search Engine That Works on Reddit Now Thanks to AI Deal

DuckDuckGo, Bing, Mojeek, and other search engines are not returning full Reddit results any more.

Archive link

47 comments

Crazy that google paid for rights after reddit's gone to shit. 3 years ago it would have been great, but these days I can't remember the last time I clicked on a reddit link for an answer.
- It's because the Google algo is broken, and a good percentage of Internet users search "thing you want to know reddit" for everything now.
  
  Would be neat if lemmy could form similar niche communities where people talk about their hobbies and scientific interests.
  
  And it's only like that because of the CIA. Around 2016 google started prioritizing MSM
- I'm starting to wonder where people even get information anymore. Are libraries making a comeback?
  
  Discord, and I'm not happy about it.
  
  I'm starting to wonder where people even get information anymore.
  Facebook, Twitter, and other cesspools.
  
  i guess we can all start using out individual imaginations again
  
  Youtube is a primary source now. A lot of people take information from some [insert cracker specialist on random topic here] specialy about social/economic topics but realy everything.
  And to be fair there is also some actualy decent and informative channels too which makes it even worse because its not like everything on the internet/YT is false or wrong its just you picking idiot grifters as your source.

we are really in need to a viable FLOSS search engine. That can do its own indexing instead of repackaging google results like searx does. Maybe the spidering could be distributed somehow so the small self hosters could benefit from it while also able to apply their own standards, priorities, sorting etc.
- unfortunately search is expensive in a way that FLOSS does not solve, it requires a lot of hosting infrastructure and boring volunteer labor to fine-tune results to combat spam (and spam might even benefit from looking at the FLOSS rules that filter it)
  
  (and spam might even benefit from looking at the FLOSS rules that filter it)
  idk i feel like that might be a problem which is created or at least greatly exaggerated by monopolies. if there was a diversity of search engines it would be much more difficult to do shitty SEO on all of them at the same time. You'd need a whole team combing through repo hosting sites and mailing lists to figure it out.
- Crawling could be distributed and shared, but indexing is a bigger problem.
  All the things you'd want to be different on some sort of federated search platform (standards, priorities, and sorting, as you say) are things that require different indexing. But the index is the big expensive part that would most need to be shared.
- https://search.marginalia.nu/ is open source, extremely good, and insanely resource efficient.
  
  Thanks for this recommendation, I hadn't heard of this. Looks promising.
  
  Oh that looks cool. Do you run an install of it or are you using the install on their main page? Are there other instances
  The hw requirements aren't prohibitive. I mean it's not nothing, maybe a few hundred upfront and then the connection. Well within reach especially with support of an existing organization who'd be willing to physically house it. I guess SSDs would be the largest part of the cost.
  an x86-64 machine, have at least 16GB of RAM, and at least 4 cores. It is designed to run on physical hardware, and will likely be very expensive to run in the cloud.
  Crawling requires a decent network connection, ideally at least 1 Gbps. 100 Mbps will work, but will be slower.
  Storage requirements are highly dependent on the size of the index, and the number of documents being indexed. For 100,000 documents, you can probably get away with 2 TB of SSD storage, and 4 TB of mechanical storage for the crawl data.
  I don't know how far 100k documents gets you. It doesn't sound like much if you are going for the whole internet but if you are curating a more narrow subset it could be enough.
  This page has their philosophy and towards the bottom of the page, links to similar projects.

death to google
death to reddit
death to america
a century of humiliation upon the first world

Wonders of capitalist innovation

Couldn't a crawler just add a bypass for reddit's robots.txt file?
- Yeah, can't they just ignore it?
  
  If you're only making a few requests, yes, but it's very easy to detect something like an indexing crawler.
- Or just do what bing did when it started and copy Google’s results directly
  
  Or maybe they could make a browser extension like Bring Back YouTube Dislikes that will send the relevant metadata of the page back for indexing.

Anyone notice that Google Reddit searches got way worse like in the past few months? Used to be able to get away with "search query" but now it often gives whatever it considers synonyms even when in quotes.
Often the first few results are just completed unrelated. Like recently it considered a proper name of a city the synonym of the word district for me lol

@yogthos@lemmygrad.ml this looks like something you would post.
- that's very much up my alley, thanks for tagging :)

Google is now the only search engine that can surface results from Reddit...
I'm terrible at proofreading so if I can nearly immediately find a mistake - the website isn't even trying.
- What is the mistake?

Lol terrible

All my homies hate reddit.

Searx is an alternative meta-search engine that can include google results. If you don't want to host an instance you can find one here: https://searx.space/
I like https://searx.work/ personally.
- A thing I found annoying about searx is that it's instances sometimes stop working, so I have to switch to other ones.
- I like the customization options for Searx, but I had to eventually switch back to Duckduckgo/Startpage. Sometimes an engine would start timing out or requesting captchas, or an instance would go down and I'd have to add a new one to my firefox desktop/mobile, or I'd notice that the results I got through an instance via one engine or another would be very different from the same search ran on said engine directly. I feel bad, but I just don't have the time or patience for it.