CPU. Have the same setup, just with a 3060ti and ryzen 9 5900X (Arch btw). More processing power is key. Made my sister upgrade to ryzen 7 5800 and she's loading much faster. Hope that helps.
*Will also note that they go on sale quite often and can snag a good one for less than 300 USD.
I suppose the CPU but it could help to understand which exact disk and which DIMMs (including latencies) you're using.
Bear in mind you can easily drop a 5800 X3D into practically any AM4 system. The large L3 somewhat lowers the significance faster DDR4 with tight timings (though it still doesn't hurt to have that too).
I believe the sweet spot for Matisse and Vermeer (zen 2 and 3 chiplet design CPUs) memory wise is 3600 CL16. Later batches of these CPUs may have better luck with stability at 3800 MT/s (due to process maturity) but that would be at a point of diminishing returns (especially with the X3D).
You might improve the performance of your current CPU by tuning your memory, but I won't know how far you can get with your kit unless we have the exact model number.
The 1600AF seems to be zen+ based (characteristically similar to the Ryzen 5 2600). For zen+, something like 3200 MT/s CL14 would be optimal if you can achieve it, 3200 CL16 would be good too.
I've tried Wayland before but had issues with various things that I can't enumerate right now. I'll give it another shot if you think it'll really help