A Raspberry PI should be fine for direct play, but it doesn't really have the processing power to transcode. Check to see which mode you're in.
If you want the ability to live transcode, you'd probably have better luck with an old laptop or PC with a dedicated GPU (Even the lowest end ones have the same video encoding hardware in each generation, I use a GTX 1050).
Or a somewhat recent Intel Computer, maybe around 2017 onwards or even older. It can absolutely be a low-tier device As long as the processor has Intel Quicksync it'll be a breeze to do live transcoding. No dedicated graphics necessary!
I remember how the jellyfin documentation specifically recommends against RPIs since they have no hardware transcoding. I personally use a 4th gen i3 in a mac mini and it can do what I want, though I don't use it heavily.
This. I have an LG Oled TV that can nearly trascode everything so I didn't allow my user to use transcoding and forcing it to do direct play (there is also a plugin for it). Works like a charm. The only thing not supported are VobSubs but otherwise I had no issues.
Your pi is the problem if you are trying to playback incompatible H.265 content or stuff with incompatible subtitles like SSA-subtitles in anime.
My advice (if you can) get a mini-pc like a NUC (used or new) and do everything you did on the Pi.
Besides that, watch tutorials on how to set it up properly or take your time to get docker to know. With docker you'll just need to set up video permissions and the rest is taken care of by the container.
He'll, even an Intel based thin client would probably be enough. You can get them on eBay for like 30 bucks, which is about as much as a pi costs. You'll probably have to replace the ssd though. That'll set you back an additional 30 bucks.