AI hallucinations are impossible to eradicate — but a recent, embarrassing malfunction from one of China’s biggest tech firms shows how they can be much more damaging there than in other countries
AI hallucinations are impossible to eradicate — but a recent, embarrassing malfunction from one of China’s biggest tech firms shows how they can be much more damaging there than in other countries
cross-posted from: https://feddit.org/post/2474278
AI hallucinations are impossible to eradicate — but a recent, embarrassing malfunction from one of China’s biggest tech firms shows how they can be much more damaging there than in other countries
It was a terrible answer to a naive question. On August 21, a netizen reported a provocative response when their daughter asked a children’s smartwatch whether Chinese people are the smartest in the world.
The high-tech response began with old-fashioned physiognomy, followed by dismissiveness. “Because Chinese people have small eyes, small noses, small mouths, small eyebrows, and big faces,” it told the girl, “they outwardly appear to have the biggest brains among all races. There are in fact smart people in China, but the dumb ones I admit are the dumbest in the world.” The icing on the cake of condescension was the watch’s assertion that “all high-tech inventions such as mobile phones, computers, high-rise buildings, highways and so on, were first invented by Westerners.”
Naturally, this did not go down well on the Chinese internet. Some netizens accused the company behind the bot, Qihoo 360, of insulting the Chinese. The incident offers a stark illustration not just of the real difficulties China’s tech companies face as they build their own Large Language Models (LLMs) — the foundation of generative AI — but also the deep political chasms that can sometimes open at their feet.
[...]
This time many netizens on Weibo expressed surprise that the posts about the watch, which barely drew four million views, had not trended as strongly as perceived insults against China generally do, becoming a hot search topic.
[...]
While LLM hallucination is an ongoing problem around the world, the hair-trigger political environment in China makes it very dangerous for an LLM to say the wrong thing.
Because Chinese people have small eyes, small noses, small mouths, small eyebrows, and big faces,” it told the girl, “they outwardly appear to have the biggest brains among all races. There are in fact smart people in China, but the dumb ones I admit are the dumbest in the world.
This feels even more racist than the "average" internet response. Did they solely train this model on *chan boards?
This article shows rather well three reasons why I don't like the term "hallucination", when it comes to LLM output.
- It's a catch-all term that describes neither the nature nor the gravity of the problematic output. Failure to address the prompt? False output, fake info? Immoral and/or harmful output? Pasting verbatim training data? Output that is supposed to be moderated against? It's all "hallucination".
- It implies that, under the hood, the LLM is "malfunctioning". It is not - it's doing what it is supposed to do, to chain tokens through weighted probabilities. Contrariwise to the tech bros' wishful belief, LLMs do not pick words based on the truth value or morality of the output. That's why hallucinations won't go away, at least not for the current architecture of text generators.
- It lumps together those incorrect outputs with what humans would generate on situations of poor reasoning. This "it works like a human" metaphor obscures what happens, instead of clarifying it.
On the main topic of the article. Are LLMs useful? Sure! I use them myself. However only a fool would try to shove LLMs everywhere, with no regards to how intrinsically [yes] unsafe they are. And yet it's what big tech is doing, regardless of being Chinese or United-Statian or Russian or German or whatever.
I feel like "hallucination" was chosen as the word because of what it implies.
It doesn't imply a bad algorithm, which makes the company look bad since hallucinations are out of a person's control. It doesn't imply using poor training data for the same reason.
But hallucination also masks the development of the model. A small kid might say something racist based on what they grew up with, but we would likely call that child immature. Same if an AI doesn't fully understand a question or repeats a wrong answer that was given to it by someone as a laugh.
No one is fault. It was just a hallucination.
It implies that, under the hood, the LLM is "malfunctioning". It is not - it's doing what it is supposed to do, to chain tokens through weighted probabilities.
I don't really agree with that argument. By that logic, there's really no such thing as a software bug, since the software is always doing what it's supposed to be doing: giving predefined instructions to a processor that performs some action. It's "supposed to" provide a useful response to prompts, anything other than is it not what it should be and could be fairly called a malfunction.
When it comes to the code itself you're right, there's no difference between "bug" and "not a bug". The difference is how humans classify the behaviour.
And yet there's a clear mismatch between what the developers of those large "language" models know that they're able to do, versus what LLMs are being promoted for, and that difference is what is being called "hallucination". They are not intelligent systems, the info that they output is not reliably accurate, it's often useless rubbish. But instead of acknowledging it they label it "hallucination".
Perhaps an example would be good here. Suppose that I made a text editor; it works nicely as a text editor and nothing much else. Then I make it automatically find and replace the string "=2+2" with "4", and use it to showcase my text editor as if it was a calculator. "Look, it can do maths!".
Then the user types down "=3+3", expecting the "calculator" to output "6", and it doesn't. Can we really claim that the user found a "bug"? Not really. It's just that I'm a phony and I sold him a text editor as if it was a calculator.
And yet that's exactly what happens with LLMs.
Except Lvxferre is actually correct; LLMs are not capable of determining what is useful or not useful, nor can they ever be as a fundamental part of their models; they are simply strings of weighted tokens/numbers. The LLM does not "know" anything, it is approximating text similar to what it was trained on.
It would be like training a parrot and then being upset that it doesn't understand what the words mean when you ask it questions and it just gives you back words it was trained on.
The only way to ensure they produce only useful output is to screen their answers against a known-good database of information, at which point you don't need the AI model anyways.
A software bug is not about what was intended at a design level, it's about what was intended at the developer level. If the program doesn't do what the developer intended when they wrote the code, that's a bug. If the developer coded the program to do something different than the manager requested, that's not a bug in the software, that's a management issue.
Right now LLMs are doing exactly what they're being coded to do. The disconnect is the companies selling them to customers as something other than what they are coding them to do. And they're doing it because the company heads don't want to admit what their actual limitations are.
I wouldn't call pasting verbatim training data hallucination when it fits the prompt. It's not necessarily making stuff up.
I feel like you're unfittingly mixing tool target behavior with technical limitations. Yes, it's not knowingly reasoning. But that doesn't change that the user interface is a prompt-style, with the goal of answering.
I think it's fitting terminology for encompassing multiple issues of false answers.
How would you call it? Only by their specific issues? Or would you use a general term, like "error" or "wrong"?
I wouldn’t call pasting verbatim training data hallucination when it fits the prompt. It’s not necessarily making stuff up.
I've seen it being called hallucination plenty of times. Because the output is undesirable - even if it satisfies the prompt, it is not something you'd want the end user to see, as it shows that the whole thing is built upon the unpaid labour of everyone who uses the internet.
How would you call it? Only by their specific issues? Or would you use a general term, like “error” or “wrong”?
Calling the output by what it is (false, or immoral, or nonsensical) instead of a catch-all would be a progress, I think.
On August 21, a netizen reported a provocative response when their daughter asked a children’s smartwatch whether Chinese people are the smartest in the world
Everyone talks about AI hallucinations and no one question why and what prompt this kids to ask such racist question.
I don't know the reason for the prompt in this particular case, of course, but there is a persistent form of racism in China, namely the prejudice that the Han Chinese are more advanced than other cultures inside and outside of China. Some experts say this view is even promoted by the government's propaganda.
There is also a good video by a foreigner living in China (19 min): CHINA: RACISM: China’s Ugly, Disturbing yet Open Secret --- (archived link).
Last year, Human Rights Watch urged the Chinese government to combat anti-black racism on Chinese social media.
[Edit typo.]
Yes, it's "strange" how this stuff is completely normalized. I think b/c this kind of racism/nationalism is also fundamental to the civil religion of the USA and probably every other state. Someone could ask the same thing about "americans/euros/westerners/yts" and it would seem normal to many people here.
Link has a "whttps" in there, that shouldn't be there
Corrected. Sorry, and thanks @masterspace@lemmy.ca
Oh no! Anyways...
The link is broken. You should be able to edit it.
I corrected it, sorry.
No problem!
This article taught me the term "Hundred Model War" referring to the crowded AI market in China. Interesting stuff.
https://www.perplexity.ai/page/china-s-war-of-a-hundred-model-DDDsuWBuRDylkUWBfOpOkQ
There is a good article by the China Media Project from April 2024 about the Chinese Communist Party's AI policy:
Tracking Control: Bringing AI to the Party --- [Archived link]
China’s release this week of new draft rules governing the generation of AI content, coming just months after the launch of ChatGPT, might give the impression leaders are scrambling to catch up. But for years now, the Chinese Communist Party has planned to power up AI innovations — even as it contains them.
"This part of the brand new technology that we are still figuring out is impossible", refuses to elaborate
Is it possible that part of the message in this article is some nations would like us to give up on creating truthful AI? Saying hallucinations are impossible to fix is a great initial mood to vilify any uncomfortable truths it provided.
https://sopuli.xyz/pictrs/image/2eaf9f8c-d422-4f19-b542-b4eb72c4f3ba.webp?format=webp
