Because people over use it. YAML is pretty good for short config files that need to be human readable but it falls apart with complex multi line strings and escaping.
I think there are much better clearly delimited for machine reading purposes formats out there that you should prefer if you're writing a really heavy config file and, tbh, I think for everything else .ini is probably "good enough".
I agree - YAML is not suitable for complex cases that people use it in, like Terraform and Home Assistant. My pet peeve is a YAML config in a situation that really calls for more abstraction, like functions and variables. I'd like to see more use of the class of configuration languages that support that stuff, like Dhall, Cue, and Nickel.
There is another gotcha which is that YAML has more room for ambiguity than, say, JSON. YAML has a lot of ways to say true and false, and it's implicit quoting is a bit complex. So some values that you expect to be strings might be interpreted as something els.
For those highly complex situations is Lua still viewed as the ideal solution? Lua is sort of legendary for game configuration and seems to strike a good expressiveness/accessibility balance for modders and the casually technical.
I think it depends. Lua is great for scripting - like when X happens do Y. I agree that makes sense for a case like Home Assistant. Sometimes you really want the result to be a data structure, not an interactive program, in which case I think more sophisticated configuration (as opposed to scripting) languages might be better.