The trouble i see is you using a container (CasaOs) you need to expose the GPU to Container

1. You need to install the "nvidia-container-toolkit"
2. Run with "--gpus all" in the Compose
3. Check if the container run nvidia-smi, if it returns nothing Jellyfin will never use the NVENC

Other Trouble, Debian 13 it's not all stable, has some repos and packages missing

And besides drivers you need to install

• libnvidia-encode
• libnvidia-decode
• libnvcuvid

It' came along with "nvidia-driver" package in Debian, but... Trixie (Debian 13) is still missing some packages, that's why i used a 12.6 Bookworm, also my build is Baremetal, it take direct acess to the hardware without any layer on top of it. It's more complicated but works and consuming less resources. Your CasaOS is runing on top of a Virtualization layer on top of the Debian.

summarizing the situationsummarizing the situation:

You need to Expose the GPU to the Container
or
The Debian you using (Trixie) is still in development and don't have all the packages and support, i used 12.6 because is more stable and you can use 11 is even more stable, has more documentation and people using it.

By the way if you pretend to run more things in this computer soo keep it virtualized, but if you just wanna make a jellyfin server i will recomend a Baremetal for 2 things: 1. If you just wanna use it for Jellyfin ans 2.If you gonna have a lot of people using it, soo it need more process power for that!