Yeah they wouldnt complain... except if theyre my grandfather who started complaining in vietnam to a hotel receptionist that they dont speak hungarian, a language about 14 million people speak in the world and the only language it has mutual intelligibility with is spoken by about 10000 people....
Wait, in that case maybe complaining is a bit justified? Different of course of it's a place off the beaten path, but if it's a hotbed of English-speaking tourists then having staff that speak their language seems pretty important.
Guess it's really hard to say without more context (story of the internet).
I don't even know why some native speakers complain about having to deal with broken English when the learner only needs to know a tiny fraction of the language to be intelligible. As long as they have the vocab, grammar is mostly optional.