At least one drawback I can see is that apparently the user has to manually opt-out some macros (the ones that aren't pure) of the caching. Which would require everyone using the rust compiler to know which macros of which crates are pure and which ones not. I guess that the ones writing the macros could do that, but then you rely on library maintainers to know about this specific optimization.
It really should be opt-in so the maintainers of libraries that know about the optimization and know that their macro is pure can annotate them as so.
Right, if there was a way to mark them as pure (const fn?) that would solve it, mostly - as long as upstream crates started annotating their proc macros as such.