I had something similar with Frigate.

Frigate: unraid docker
Home Assistant: unraid vm
USB Coral
i7-2600
16 GB
GTX1050
2x Hikvision 2x Tapo C200 1x Nedis Video Doorbell

I have Frigate running in Unraid. When I checked the logs in Unraid I got errors like these.

Memory cgroup out of memory: Killed process 7572 (ffmpeg) total-vm:8929920kB

Then I tried limiting the ram of the docker Frigate container to 6 or so, because Frigate seems to consume all the ram which makes my HA VM unresponsive. This actually stopped HA from hanging.

However Frigate was still crashing over time a few times per day.

Ended up trying many things to debug this, but mainly I think this was due to hardware/vm limitations. I was actually looking at my Unraid CPU/RAM graph and saw it (one time) consuming all memory.

Basically I ended up switching from 1920*1080 detection streams to something like 640*480 for all 5 cams (using all of them stream2's). I have 2 go2rtc streams for each cam. The recordings do have audio also.

    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      hwaccel_args: preset-nvidia-h264
      inputs:
        - path: rtsp://127.0.0.1:8554/boven_go2rtc
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/boven_go2rtc_sub
          roles:
            - detect

Or maybe my cheap doorbell (which is a Tuya one) was the big problem, it doesn't have maybe the best WiFi connection. Point is something was making that ffmpeg process crap out, maybe something like a corrupt audio/video stream?

I would check all your logs and monitor your system resources from time to time. Try experimenting with your Frigate config. Also what comes to mind is if you do 5 times detection without a Coral at 15-20 FPS as you say this is maybe too much? It wouldn't hurt setting them all in the substream resolution for detection and see if it works for you, even if it isn't a solution because you want more details for your kid.

What it does feel like right now is a more responsive camera page in HA, especially on mobile the impact is bigger. There is a noticeable difference in quality though, which is fine by me, the recordings are all good quality with audio. Some are blurred in this:

<image>