That’s true but they usually spoof some person I don’t know. So if I google it and it’s some Betty from East Bumfuck, I know not to pick up. Now, if they spoofed someone I know, that’s different.
The thing is, there are legit reasons to want to spoof numbers. My use case is that if I'm not reachable via sip I send the call over the normal phone network and set cid to the calling number.
I guess a middle ground would be that you could spoof numbers to phone numbers you've registered and verified with your provider.
To be honest though, the scammers and spammers will always find a way round it and spoof anyway.