Dangerous? Signal Blasts Google Effort to Use AI to Scan for Scam Phone Calls
Dangerous? Signal Blasts Google Effort to Use AI to Scan for Scam Phone Calls
One of the leaders of the private messaging app Signal calls the feature 'dangerous' because it could potentially spy on users.
Google’s AI model will potentially listen in on all your phone calls — or at least ones it suspects are coming from a fraudster.
To protect the user’s privacy, the company says Gemini Nano operates locally, without connecting to the internet. “This protection all happens on-device, so your conversation stays private to you. We’ll share more about this opt-in feature later this year,” the company says.
“This is incredibly dangerous,” says Meredith Whittaker, the president of a foundation for the end-to-end encrypted messaging app Signal.
Whittaker —a former Google employee— argues that the entire premise of the anti-scam call feature poses a potential threat. That’s because Google could potentially program the same technology to scan for other keywords, like asking for access to abortion services.
“It lays the path for centralized, device-level client-side scanning,” she said in a post on Twitter/X. “From detecting 'scams' it's a short step to ‘detecting patterns commonly associated w/ seeking reproductive care’ or ‘commonly associated w/ providing LGBTQ resources' or ‘commonly associated with tech worker whistleblowing.’”
"...locally on device without connecting to the internet"
How would it then report such behavior to Google, without internet?
If it notifies the end user, what good does that do? My phone is at my ear, I don't stop a conversation when another app sends a notification while I'm on a call.
This will 100% report things in the background to Google.
You're putting a very large amount of trust on something which may simply require the flip of a switch to add the specified information to be sent back to Google along with all the regular heavy telemetry already feeding back...
Mega hot take on this site: I have no trust in Google
There are a few ways this could work, but it hardly seems worth the effort if it's not phoning home.
They could have an on-device database of red flags and use on-device voice recognition against that database. But then what? Pop up a "scam likely" screen while you're already mid-call? Maybe include an option to report scams back to Google with a transcript? I guess that could be useful.
Any more more than that would be a privacy nightmare. I don't want Google's AI deciding which of my conversations are private and which get sent back to Google. Any non-zero false positive rate would simply be unacceptable.
Maybe this is the first look at a new cat and mouse game: AI to detect AI-generated voices? AI-generated voice scams are already out there in the wild and will only become more common as time goes on.
My bet is it will work like their federated text prediction in gboard.
How would it then report such behavior to Google, without internet?
It doesn't
In a demo, the tech giant simulated a scam call involving a fraudster impersonating a bank. A pop-up message appeared, encouraging the user to hang up.
If it notifies the end user, what good does that do?
You can't see why it might be helpful for a user to know that they're speaking to a scammer?
I assume it means the "AI" bit is running locally (for cost/efficiency reasons and so your actual voice isn't uploaded) the results are then uploaded wherever (which is theoretically better but still hugely open to abuse)
Blasting is all fine and good but they should've slammed them really.
Or spanked
Let's talk about wiretapping laws and states where two-party consent is required to record a call.
Where I live, I must notify the other party that I am recording. If not, it's illegal. Also, any audio recorded without consent is not admissible in court.
"Not admissible because it was illegally captured" didn't give me the warm-and-fuzzies this comment sounds like it should've.
Complete tangent but what is two-party consent even for? I can imagine it gets in the way of getting a lot of evidence in cases of domestic abuse or organized crime.
Caller: [threatens me for 45 seconds] Me: "Could you call me again and repeat all that, for the recording?" Caller: [hangs]
Means both parties have to agree to be recorded (usually at the onset of a call).
"Be advised, this call is recorded for quality assurance purposes" at which point you could hang up. The 4th Amendment still applies in America, regardless of what local cops and prosecutors believe. You have a right to privacy in two-party conversations.
One of the things they glided around was whether a lot of this on-device stuff needs a special processor chip with AI+security to work?
The Pixel phones (especially newer ones) made by Google have them, but the vast majority of Android phones don't.
So either these features only work on latest Google phones (which will piss off licensees and partners), or they're using plain old CPU/GPUs to do this sort of detection, in which case it will be sniffable by malicious third-parties.
And let's not forget that if the phone can listen to your conversation to detect malicious intent, any country can legally compel Google to provide them with the data by claiming it is part of a law-enforcement investigation.
Things are going to get spicy in Android-land.
It's not just google who have AI stuff built into their phones. All recent SoCs I have seen have had NPUs going back the last couple generations. A lot of older or cheap phones won't have one, but the new devices will.
I don't see the problem with using the phones normal GPU. This shouldn't be more insecure than making a call currently is. I am pretty sure android phones don't have a secure enclave just for making calls as you can give different apps access to calling features, and most calls I make are through third party apps anyway, not via POTS. That being said android is pretty secure anyway provided you don't give permissions to the wrong app. It's more secure than your average Linux system, as each app has its own user and is only allowed to access things it has explicit permissions to access. Secure enclaves aren't all that in my opinion.
And let's not forget that if the phone can listen to your conversation to detect malicious intent, any country can legally compel Google to provide them with the data by claiming it is part of a law-enforcement investigation.
The point of doing it locally is the audio never gets sent to google directly. That being said they could definitely do some dodgy things by training the ML model to search for words like abortion, drugs, transgender, etc depending on what the laws are in the country the phone is being used in.
As a OpenSource app with no need for centralized server it would be great. I want that. As spyware configured out of my control absolutely not.
I think you do need some central server, and then check if a lot of users report certain number blocks for spam in a short amount of time. No need for AI on this one. Isn't that how most phone spam blockers work?
I have that now, its OK. But spammers cycle through numbers so quick that good amount get through
That is how it already works. I don't think most people have as much of a problem with that as complete client side screening.
It might be a good feature for the elderly as long as it's local and optionally enabled (especially if it can be enabled only for unknown callers).
Yes, I understand you would never really know if it's not always enabled. But then again, you currently don't know if anything similar isn't already enabled.
For other users, again potentially useful if it's opt in. However, many people (myself included) simply don't answer the phone anymore unless it's a caller we already know. I use Google's call screening feature for any other caller not in my contact list already, and I would estimate about 1 in 20 or 5% of such calls I receive aren't spam (marketing or fraud). Of those non-spam calls, the majority are appointment reminders I don't need.
So would I turn this feature on? No, I don't have a need. Could it be beneficial for the elderly? Yes, but probably not implemented in a way where it would actually be effective.
Honestly, if it was transparent enough, I'd be fine with a service you could turn on that would listen to the first x seconds/minutes of a phone call to try and detect an AI generated voice. I have family that would 100% wire thousands of dollars if someone with their kid's voice asked them to.
Are the elderly a new pillar of the https://en.wikipedia.org/wiki/Four_Horsemen_of_the_Infocalypse ?
Did the blast with cum??
Top 10 Cum Blasts You WONT BELIEVE (subscrab)
#1: 120 mm Penetration Cum Blast (PCB) - https://www.drdo.gov.in/drdo/120-mm-penetration-cum-blast-pcb-and-thermobaric-tb-ammunition-mbt-arjun
Checkmate
Can't really get behind her on this one. Assuming it works the way Google says it does, it couldn't be used for any of those things she's concerned about.
If it doesn't work that way, well then your entire device is compromised to begin with, because Google controls the entire operating system.
Assuming it works the way Google says it does,
There was a time when I trusted Google to be open and honest.
But that time is long in the past.
If it doesn't work that way, well then your entire device is compromised to begin with, because Google controls the entire operating system.
I feel like this would require cooperation from the manufacturer, as Google doesn't actually provide the Phone app (except when they are the manufacturer).
except that it does. some OEMs like Samsung serve their own (or at least Samsung used to; not sure if it's still true), but it's definitely available on non-Google-branded phones.
They seem to be offering a lot of this sort of thing as a value-add for buying a Pixel
They might be taking about Google Fi.