Yeah this smells like a bug in Caddy or something. I agree to try nginx or something else to see if it’s Caddy or if it’s something with the configuration of the host. The only thing I could think of is if caddy isn’t caching DNS responses and maybe is getting rate limited so it appears slower while it’s waiting on the DNS request but I am shooting in the dark as I haven’t spent much time with caddy.
I mentioned above- the subdomains were using HTTP/3, and the root entry does not. I don't know if it's something I have mis-configured or just HTTP/3 being new and maybe buggy. Either way, i disabled it globally and performance is the same.