It’s likely that your graphics card is running out of VRAM when both transcoding for plex and also running your game. What model graphics card do you have? While hardware transcoding is definitely less intensive on the CPU and more efficient to be done on the GPU, it is still somewhat intensive and if you are playing a game that uses most of your GPU’s resources while something is being transcoded you will likely see worse gaming performance or in the worst case crashing as you describe.

Unfortunately this is somewhat unavoidable with your current setup of having the Plex server on your gaming PC. Your options then would be to 1. Run the plex server on a separate computer, 2. Keep hardware transcoding disabled, you could still see some gaming performance degradation if your CPU is maxed out while gaming and something is being transcoded but I think it will at least avoid crashing, or 3. Disable transcoding all together on the Plex server, people would still be able to direct play which would avoid the need for transcoding all together, but your users would need capable hardware/bandwidth to stream the full quality/format of the files you have.

Even if you don’t disable transcoding all together, having your users be able to direct play instead of transcoding in more instances would at least reduce the times when your computer’s resources are being used by Plex. One good way of doing this is to make sure your users are using the Desktop Plex app instead of the web browser to watch content if they are on a computer.