Nothing in the thought experiment says that the program doesn't behave that way. If the program really seems like it understands language to an outside observer, you would assume it did learn language that way.
It's always only going to be useful for things like buying drugs, cases where you want to skirt regulation or you really want privacy. Which is fine. It can have it's niche. Pretending it was ever going to be more than that was a mistake.
They got the idea from Rishi Sunak