This point is really stupid and overcooked. Even if it’s true which I don’t think it is, who cares? So what? What does it even mean? It’s such an obvious and dull example of whataboutism. It’s like telling a man who’s just lost a foot that vastly more people over the course of history have lost an arm. Why would you bring that up, even if it’s true which it probably isn’t?
The distinction of material conditions being good or bad doesn't alter this arrangement. Ideals reflect and justify those material conditions, not the other way around. It's the material conditions that form the basis for ideals, whether the conditions are judged to be good or bad is an aspect of this.
My understanding was that idealism has to take into consideration the material conditions. That does not mean idealism, if properly adapted, cannot affect material conditions in turn.