I am guilty of reinventing the wheel on almost every project. It brings immense control but doubles the workload. I do this because I have trust issues, but at least in the end I have "homemade everything"
As long as you don't insist on using them even after it became clear that the off the shelf version is better in ever way and you'll never have enough time to reach its quality level.
I feel like doubling the workload is better than quadrupling the size of the project inheriting a bevy of features and tools you likely won't touch at all. Sure it's stripped out later (ideally), but I like less bloat and that includes during dev when I might have to dig through 3rd party code with its own conventions and standards packed into a 'source available' library with potentially dogshit or absent documentation.
Unused code is stripped out by the compiler, but will your homemade library properly use all the fancy instruction set extensions for matrices? IIRC it's not as simple as just compiling for the correct microarch. But I could be wrong.