Cloudflare announces AI Labyrinth, which uses AI-generated content to confuse and waste the resources of AI Crawlers and bots that ignore “no crawl” directives.

Should have called it "Black ICE".
- Definitely falls under the category of a Trap ICE card.

I have no idea why the makers of LLM crawlers think it's a good idea to ignore bot rules. The rules are there for a reason and the reasons are often more complex than "well, we just don't want you to do that". They're usually more like "why would you even do that?"
Ultimately you have to trust what the site owners say. The reason why, say, your favourite search engine returns the relevant Wikipedia pages and not bazillion random old page revisions from ages ago is that Wikipedia said "please crawl the most recent versions using canonical page names, and do not follow the links to the technical pages (including history)". Again: Why would anyone index those?
- Because you are coming from the perspective of a reasonable person
  These people are billionaires who expect to get everything for free. Rules are for the plebs, just take it already

People complain about AI possibly being unreliable, then actively root for things that are designed to make them unreliable.
- I find this amusing, had a conversation with an older relative who asked about AI because I am "the computer guy" he knows. Explained basically how I understand LLMs to operate, that they are pattern matching to guess what the next token should be based on a statistical probability. Explained that they sometimes hallucinate, or go of on wild tangents due to this and that they can be really good at aping and regurgitating things but there is no understanding simply respinning fragments to try to generate a response that pleases the asker.
  He observed, "oh we are creating computer religions, just without the practical aspects of having to operate in the mundane world that have to exist before a real religion can get started. That's good, religions that have become untethered from day to day practical life have never caused problems for anyone."
  Which I found scarily insightful.
  
  Oh good.
  now I can add digital jihad by hallucinating AI to the list of my existential terrors.
  Thank your relative for me.
- Here's the key distinction:
  This only makes AI models unreliable if they ignore "don't scrape my site" requests. If they respect the requests of the sites they're profiting from using the data from, then there's no issue.
  People want AI models to not be unreliable, but they also want them to operate with integrity in the first place, and not profit from people's work who explicitly opt-out their work from training.
  
  I'm a person.
  I dont want AI, period.
  We cant even handle humans going psycho. Last thing I want is an AI losing its shit due from being overworked producing goblin tentacle porn and going full skynet judgement day.
  Got enough on my plate dealing with a semi-sentient olestra stain trying to recreate the third reich, as is.
- This will only make models of bad actors who don't follow the rules worse quality. You want to sell a good quality AI model trained on real content instead of other misleading AI output? Just follow the rules ;)
  Doesn't sound too bad to me.
- i mean this is just designed to thwart ai bots that refuse to follow robots.txt rules of people who specifically blocked them.
- Maybe it will learn discretion and what sarcasm are instead of being a front loaded google search of 90% ads and 10% forums. It has no way of knowing if what it’s copy pasting is full of shit.

I’m imagining a sci-fi spin on this where AI generators are used to keep AI crawlers in a loop, and they accidentally end up creating some unique AI culture or relationship in the process.

I guess this is what the first iteration of the Blackwall looks like.
- Gotta say "AI Labyrinth" sounds almost as cool.

"I used the AI to destroy the AI"
- And consumed the power output of a medium country to do it.
  Yeah, great job! 👍
  
  We truly are getting dumber as a species. We're facing climate change but running some of the most power hungry processers in the world to spit out cooking recipes and homework answers for millions of people. All to better collect their data to sell products to them that will distract them from the climate disaster our corporations have caused. It's really fun to watch if it wasn't so sad.
- We had to kill the internet, to save the internet.
  
  We have to kill the Internet, to save humanity.

Surprised at the level of negativity here. Having had my sites repeatedly DDOSed offline by Claudebot and others scraping the same damned thing over and over again, thousands of times a second, I welcome any measures to help.
- I think the negativity is around the unfortunate fact that solutions like this shouldn't be necessary.

Burning 29 acres of rainforest a day to do nothing
- Bitcoin?

Imagine how much power is wasted on this unfortunate necessity.
Now imagine how much power will be wasted circumventing it.
Fucking clown world we live in
- !TooDumbToImagine@europe.pub
- On on hand, yes. On the other...imagine frustration of management of companies making and selling AI services. This is such a sweet thing to imagine.
  
  My dude, they'll literally sell services to both sides of the market.
  
  I just want to keep using uncensored AI that answers my questions. Why is this a good thing?

Considering how many false positives Cloudflare serves I see nothing but misery coming from this.
- Lol I work in healthcare and Cloudflare regularly blocks incoming electronic orders because the clinical notes "resemble" SQL injection. Nurses type all sorts of random stuff in their notes so there's no managing that. Drives me insane!
- In terms of Lemmy instances, if your instance is behind cloudflare and you turn on AI protection, federation breaks. So their tools are not very helpful for fighting the AI scraping.
  
  Can't you configure exceptions for behaviours?

And soon, the already AI-flooded net will be filled with so much nonsense that it becomes impossible for anyone to get some real work done. Sigh.
- Some of us are only here to crank hog.
  
  AROOO!

I'm glad we're burning the forests even faster in the name of identity politics.
- Do you identify as retarded?
- What has this anything to do with identity politics?
  
  Don't feed the troll

You have Thirteen hours in which to solve the labyrinth before your baby AI becomes one of us, forever.
- While AI David Bowie sings you rock lullabies.

So the web is a corporate war zone now and you can choose feudal protection or being attacked from all sides. What a time to be alive.
- There is also the corpo verified id route. In order to avoid the onslaught of AI bots and all that comes with them you'll need to sacrifice freedom, anonymity, and privacy like a good little peasant to prove you aren't a bot.. and so will everyone else. You'll likely be forced to deal with whatever AI bots are forced upon you while within the walls but better an enemy you know I guess?

Not exactly how I expected the AI wars to go, but I guess since we're in a cyberpunk world, we take what we get
- Next step is an AI that detects AI labyrinth.
  It gets trained on labyrinths generated by another AI.
  So you have an AI generating labyrinths to train an AI to detect labyrinths which are generated by another AI so that your original AI crawler doesn't get lost.
  It's gonna be AI all the way down.
  
  All the while each AI costs more power than a million human beings to run, and the world burns down around us.
  
  LLMs tend to be really bad at detecting AI generated content. I can’t imagine specialized models are much better. For the crawler, it’s also exponentially more expensive and more human work, and must be replicated for every crawler since they’re so freaking secretive.
  I think the hosts win here.
  
  Yeah that’s how we achieve singularity.

That's just BattleBots with a different name.
- No, it is far less environmentally friendly than rc bots made of metal, plastic, and electronics full of nasty little things like batteries blasting, sawing, burning and smashing one another to pieces.
- You're not wrong.
  
  Ok, I now need a screensaver that I can tie to a cloudflare instance that visualizes the generated "maze" and a bot's attempts to get out.
- They should program the actions and reactions of each system to actual battle bots and then televise the event for our entertainment.
  
  Then get bored when it devolves into a wedge meta.

this is some fucking stupid situation, we somewhat got a faster internet and these bots messing each other are hogging the bandwidth.
- nothing can be improved while capitalism or authority exist; all improvement will be seized and used to oppress.
  
  The problem you aren’t recognizing is that, until humans are no longer driven by self preservation, there will always be oppression in any system. They all have and will continue to breakdown. It’s easy to blame capitalism but even socialist systems eventually cave under the weight of greed and power. We are the problem mon frère.
  
  How can authority not exist? That's staggeringly broad
  
  That's not really relevant here. This is more of a "genie is out of the bottle and now we have to learn how to deal with it situation". The idea and technology of bots and AI training already exists. There's no socioeconomic system that is going to magically make that go away.
  
  Lost on Lemmy?
- Lol website traffic accounts for like 1% of bandwidth budget. 1 netflix movie is like 20k web pages.
- Especially since the solution I cooked up for my site works just fine and took a lot less work. This is simply to identify the incoming requests from these damn bots -- which is not difficult, since they ignore all directives and sanity and try to slam your site with like 200+ requests per second, that makes 'em easy to spot -- and simply IP ban them. This is considerably simpler, and doesn't require an entire nuclear plant powered AI to combat the opposition's nuclear plant powered AI.
  In fact, anybody who doesn't exhibit a sane crawl rate gets blocked from my site automatically. For a while, most of them were coming from Russian IP address zones for some reason. These days Amazon is the worst offender, I guess their Rufus AI or whatever the fuck it is tries to pester other retail sites to "learn" about products rather than sticking to its own domain.
  Fuck 'em. Route those motherfuckers right to /dev/null.
  
  and try to slam your site with like 200+ requests per second
  Your solution would do nothing to stop the crawlers that are operating 10ish rps. There's ones out there operating at a mere 2rps but when multiple companies are doing it at the same time 24x7x365 it adds up.
  Some incredibly talented people have been battling this since last year and your solution has been tried multiple times. It's not effective in all instances and can require a LOT of manual intervention and SysAdmin time.
  https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/
  
  Geez, that's a lot of requests!
  
  the only problem with that solution being applied to generic websites is schools and institutions can have many legitimate users from one IP address and many sites don't want a chance to accidentally block one.
  
  Cloudflare offers that too, but you can't always tell
- It's what I've been saying about technology for the past decade or two .... we've hit an upper limit to our technological development ... that limit is on individual human greed where small groups of people or massively wealthy people hinder or delay any further development because they're always trying to find ways to make money off it, prevent others from making money off it, monopolize an area or section of society .... capitalism is literally our world's bottleneck and it's being choked off by an oddly shaped gold bar at this point.

Generating content with AI to throw off crawlers. I dread to think of the resources we’re wasting on this utter insanity now, but hey who the fuck cares as long as the line keeps going up for these leeches.

So the world is now wasting energy and resources to generate AI content in order to combat AI crawlers, by making them waste more energy and resources. Great! 👍
- The energy cost of inference is overstated. Small models, or “sparse” models like Deepseek are not expensive to run. Training is a one-time cost that still pales in comparison to, like, making aluminum.
  Doubly so once inference goes more on-device.
  Basically, only Altman and his tech bro acolytes want AI to be cost prohibitive so he can have a monopoly. Also, he’s full of shit, and everyone in the industry knows it.
  AI as it’s implemented has plenty of enshittification, but the energy cost is kinda a red herring.

Why do I have the feeling that I will end up in that nightmare with my privacy focused and ad-free Browser setup. I already end up in captcha hell too often because of it.
- Of course it will. Cloudflare has already ruined the web and it's just another step further.

So we're burning fossil fuels and destroying the planet so bots can try to deceive one another on the Internet in pursuit of our personal data. I feel like dystopian cyberpunk predictions didn't fully understand how fucking stupid we are...
- They probably knew, but the truth is just boring and it's funner to dramatize things, haha.

Will this further fuck up the inaccurate nature of AI results? While I'm rooting against shitty AI usage, the general population is still trusting it and making results worse will, most likely, make people believe even more wrong stuff.
- The article says it's not poisoning the AI data, only providing valid facts. The scraper still gets content, just not the content it was aiming for.
  E:
  It is important to us that we don’t generate inaccurate content that contributes to the spread of misinformation on the Internet, so the content we generate is real and related to scientific facts, just not relevant or proprietary to the site being crawled.
  
  Thank you for catching that. Even reading through again, I couldn't find it while skimming. With the mention of X2 and RSS, I assumed that paragraph would just be more technical description outside my knowledge. Instead, what I did hone in on was
  "No real human would go four links deep into a maze of AI-generated nonsense."
  Leading me to be pessimistic.
  
  if you're dumb enough to trust a large language model because someone told you "iTs Ai!" no amount of facts will be of great utility to you.
  
  and the data for the LLM is now salted with procedural garbage. it's great!
  
  Until the AI generating the content starts hallucinating.
- If you're dumb enough and care little enough about the truth, I'm not really going to try coming at you with rationality and sense. I'm down to do an accelerationism here. fuck it. burn it down.
  remember; these companies all run at a loss. if we can hold them off for a while, they'll stop getting so much investment.
  
  The problem I see with poisoning the data is the AI's being trained for law enforcement hallucinating false facts used to arrest and convict people.

So they rewrote Nepenthes (or Iocaine, Spigot, Django-llm-poison, Quixotic, Konterfai, Caddy-defender, plus inevitably some Rust versions)
Edit, but with ✨AI✨ and apparently only true facts
- Cloudflare is providing the service, not libraries
- It's the consequences of the MIT and Apache licenses showing up in real time.
  GPL your software, people!

hey look it's that "zip bomb" I mentioned.
fuck cloudflare though.

Relevant excerpt from part 11 of Anathem (2008) by Neal Stephenson:
- One of my favorite books! Great world building and quite thought provoking!
- Read Anathema last year, really enjoyed it!

Damned ~~Arasaka~~Cloudflare ice walls are such a pain

I swear someone released this exact thing a few weeks ago
- We want names
  
  https://www.404media.co/developer-creates-infinite-maze-to-trap-ai-crawlers-in/

while allowing legitimate users and verified crawlers to browse normally.
What is a "verified crawler" though? What I worry about is, is it only big companies like Google that are allowed to have them now?
- I assume a crawler which adheres to robots.txt
  
  I would love to think so. But the word "verified" suggests more.
- I dunno. I don't find any sympathy with any of these fuckers though. this is not a generally useful technology, it is not something the average person ever needs to see, and honestly, just fuck em. Fuck anyone messing with open source to engorge the garbage dispenser.
  
  Any accessibility service will also see the "hidden links", and while a blind person with a screen reader will notice if they wonder off into generated pages, it will waste their time too. Especially if they don't know about such "feature" they'll be very confused.
  Also, I don't know about you, but I absolutely have a use for crawling X, Google maps, Reddit, YouTube, and getting information from there without interacting with the service myself.

Cloudflare kind of real for this. I love it.
It makes perfect sense for them as a business, infinite automated traffic equals infinite costs and lower server stability, but at the same time how often do giant tech companies do things that make sense these days?
- Kind of seems like they simply installed this dude's tarpit from a few months ago
  https://zadzmo.org/code/nepenthes/
- I'm sorry how does it make sense?

Be great if these reinforced facts.
Earth us an imperfect oblate spheroid.
Humans landed on moon.
Taiwan is an independent nation.
Edit: incorporated better information
- *imperfect oblate spheroid
  
  Thank you. I have updated my post.

I introduce to you, the Trace Buster Buster!
https://youtu.be/Iw3G80bplTg
If you've never seen the movie The Big Hit, it's great.

I am not happy with how much internet relies on cloudflare. However, they have a strong set of products

DNA Lounge has something similar - I think they even mentioned infinite JavaScript loops, and images that expand like zip-bombs.

Will it actually allow ordinary users to browse normally, though? Their other stuff breaks in minority browsers. Have they tested this well enough so that it won't? (I'd bet not.)

So this showed up last week: https://github.com/raminf/RoboNope-nginx
Similar vibe, minus the AI.

Now this is a AI trap worth using. Don't waste your money and resources hosting something yourself, let Cloudflare do it for you if you don't want AI scraping your shit.

Spiderman pointing at Spiderman meme.

And this, ladies and gentleman, is how you actually make profits on AI.