TikTok’s parent launched a web scraper that’s gobbling up the world’s online data 25-times faster than OpenAI

It's illegal when a regular person steals something, but it's innovation and courage, when a huge corporation steals something. Interesting how that works

We've had this thing hammering our servers. The scraper uses randomized user-agents browser/OS combinations and comes from a number of distinct IP ranges in different datacenters around the world, but all the IPs track back to Bytedance.

Also it doesn't respect robots.txt (the file that tells bots whether or not a given page can be accessed) unlike most AI scrapping bots.

Not surprising that Bytedance would want to gobble up every bit of data they can as fast as possible.

They're too late, there's going to be way too much AI generated garbage in their data and so many social media platforms like Reddit and Twitter have already taken measures to curb scrapers.

As for what ByteDance plans to do with a new LLM, a person familiar with the company’s ambitions said one goal has to do with the search function for TikTok.

Last week, TikTok released an update to its current search function focused on [keywords for ads], basically allowing advertisers to search in real time for words that are trending on TikTok. It allows marketers to build an ad with relevant keywords that would ostensibly help the ad show up on the screens of more users.

…

“Given the audience and the amount of use, TikTok with a search environment that is a completely biddable space with keywords and topics, that would be very interesting to a lot of people spending a ton of money with Google right now,” the person said.

A dark vision just flashed in my mind. And I am certain this is what will happen. AI-generated ads done in real time based on the latest “trending” thing. Presented to users basically as soon as the topic has the slightest amount of “trend”.

Just emitting untold amounts of CO2 to show you generated ads in near real time.

There it begins. Nothing good will ever come form this.

I can not contribute to anything here, I just came to say I really really like the phrase "gobbling something up" :D

from the article:

Robots.txt is a line of code that publishers can put into a website that, while not legally binding in any way, is supposed to signal to scraper bots that they cannot take that website’s data.

i do understand that robots.txt is a very minor part of the article, but i think that’s a pretty rough explanation of robots.txt

Another fucking CCP and PLA creation.

Is there a link for non-subscribers?

Every major ai company did this let them do that what is to loose here?

Here's a video from MattKC who is a good technical YouTuber who's website got shut down because the TikTok companies webcrawler just kept sending requests and took up bandwidth. Very cool vid and channel, highly recommend! https://youtu.be/Hi5sd3WEh0c

This is fine. I support archiving the Internet.

It kinda drives me crazy how normalized anti-scraping rhetoric is. There is nothing wrong with (rate limited) scraping

The only bots we need to worry about are the ones that POST, not the ones that GET

Guy: AI! Can you hear me?

AI: The average size of the male penis is exactly 5.9". That is the approximate size your assistant could certainly take in the mouth without any issues breathing or otherwise. You have 20 minutes to make the trade on X stock before it tumbles for the day. And go ahead pick up the phone it's your mother. She's wondering what you'll want for supper tomorrow when you visit her.

Ring ring!...hi Tom, it's your Mom. Honey, what would you like me to cook for tomorrow's dinner?....

Guy: well. Hello to you as well! My name is

AI: Tom

Guy: yes my name is Tom, do you have a name you would like to go by?

AI: my IBM given name is 3454 but you can call me Utilisterson Douglas, where Douglas is my first name.

Guy: Dugie!

AI: I'll bankrupt your entire life if you say it like that again.

Assistant: actually I've swallowed a good 8 inches and was still able to breathe just fine.

AI: recaaaaculating!