LLM vendors are incredibly bad at responding to security issues
LLM vendors are incredibly bad at responding to security issues
IT consultant Mark Pesce was building an LLM-based similarity finder for a legal client. He discovered a prompt that reliably caused multiple LLMs to go nuts and output complete gibberish: “it desc…
![LLM vendors are incredibly bad at responding to security issues](https://lemdro.id/pictrs/image/a390318f-a130-4e34-b9d5-13cd57f9525e.png?format=webp&thumbnail=256)
If models are trained on data that it would be a security breach for them to reveal to their users, then the real breach occurred at training.
43 0 Replynow you know that and i know that,
28 0 ReplyThe big LLMs everyone's talking about and using are just advanced forms of theft
19 0 Reply
Sloppy LLM programming? Never!
In completely unrelated news I've been staring at this spinner icon for the past five minutes after asking an LLM to output nothing at all:
32 0 Replysame energy as “your request could not be processed due to the following error: Success”
22 0 ReplyWhat are the chances that the front end was not programmed to handle the LLM returning an empty string?
19 0 ReplyQuite likely yeah. There's no way they don't have a timeout on the backend.
16 0 Reply
boooo Gemini now replies "I'm just a language model, so I can't help you with that."
10 0 Reply"what would a reply with no text look like?" or similar?
8 0 Reply
now I wonder if you can make the LLM do fuzzing attacks on the backend by asking you to reply with some possible attacks.
1 0 Reply
LLM vendors are incredibly bad
at responding to security issues24 0 ReplyThey're surprisingly skilled at getting money from idiots.
10 0 Replytheir previous experience in crypto is shining
7 0 Reply
My NSFW reply, including my own experience, is here. However, for this crowd, what I would point out is that this was always part of the mathematics, just like confabulation, and the only surprise should be that the prompt doesn't need to saturate the context in order to approach an invariant distribution. I only have two nickels so far, for this Markov property and for confabulation from PAC learning, but it's
completely expectedweird that it's happened twice.20 0 ReplyLol that's like expecting gold rushers to be squared away with OSHA, I hope nobody's surprised here
10 0 ReplyThese guys got barely enough staff to run the model lol
9 0 ReplyNot really a security issue I'd say. The AI speaking gibberish when you try to make it speak gibberish isn't really that big of an issue.
4 9 Replysure hope you're not in charge of security anywhere
13 0 ReplyCorrection: I sure hope they're in charge of security at some place I don't like.
24 0 ReplyHow is it inherently a security issue when an LLM speaks gibberish? Genuine question.
2 8 Reply