I've used both plenty and only once I thought Caddy was harder: caching. It requires you to install a plugin that also doesn't have the easiest of configs. I think there's a new and simpler one nowadays, but haven't tried it yet.
I now use Caddy by default for everything new I make/host.