Earthlings don't eat earth because "earth" isn't something to be eaten. Sperm whales don't eat sperm because sperm is (usually) not something to be eaten.
Fruit, however, is supposed to be eaten, and by many, many animals. By calling something a fruit fly, the expectation for most people would be that its diet consists mainly of fruit
Speak for yourself, I rather like potatoes, or as the French call them 'pomme de terre' which if I recall would literally translate as 'apple of earth'.