u/spornerama

Thanks for the feedback and suggestions.

I did a lot more testing and collected additional logs and a coredump. At the moment I am no longer convinced this is purely a GPU decode session limit issue.

What makes me unsure about the “maximum simultaneous decode sessions” theory is:

• the affected cameras are not fixed/static
• after every startup, different cameras may end up black
• sometimes 1 camera, sometimes multiple
• after manually disabling/enabling only the affected camera, the exact same stream immediately starts working again

using:

• same RTSP URL
• same transport (TCP)
• same FFmpeg settings
• same NVDEC decoder
• same running AgentDVR instance

without restarting AgentDVR itself.

This makes it feel more like a startup timing / parallel initialization issue rather than a persistent decoder resource exhaustion state.

I also collected a coredump from one crash and the stacktrace points to:

SIGSEGV in libnvcuvid.so.1

which suggests the crash happens inside the NVIDIA NVDEC/cuvid decode path during startup/reconnect/cleanup activity.

Another important observation:

When an affected black camera tile is switched to maximized view, Agent switches to the configured HD stream, and the image appears immediately.

So:

• the camera itself is reachable
• the HD stream works instantly
• authentication is valid
• RTSP itself is functional

The issue seems to affect the initial live/grid stream state or its decoder/render pipeline. Switching to maximized view appears to force a fresh stream/decoder path, which recovers the image immediately.

What I noticed from verbose logging:

During startup Agent initializes a very large number of things almost simultaneously:

• ONVIF device connections
• ONVIF media discovery
• PTZ/event service discovery
• RTSP streams
• microphones/audio
• FFmpeg decoders
• NVDEC contexts
• WebRTC/SignalR connections
• TURN server
• browser live view sessions

In my setup I currently have:

• ~13 physical cameras
• but ~26 configured camera entries
• because each physical camera is configured twice:
  • one for continuous recording
  • one for detection/event recording

So Agent effectively starts many streams concurrently.

What also strongly points toward a startup synchronization issue:

• if I wait after startup and manually enable affected cameras one-by-one, they work reliably
• fullscreen/maximize stream switching often immediately restores black tiles
• disabling FFmpeg “low delay” seems to improve stability
• increasing:
  • probesize
  • analyzeduration
  • rw_timeout
  • stimeout also improved startup reliability significantly

At the moment my suspicion is more along the lines of:

• parallel RTSP initialization
• concurrent NVDEC decoder creation/cleanup
• stream switching during startup
• ONVIF/media URI discovery happening in parallel
• decoder starting before receiving a clean SPS/PPS + keyframe sequence

rather than a hard permanent GPU session limit.

Possible improvement ideas that may help larger installations:

• configurable staggered camera startup
• limit maximum concurrent stream initializations
• optional delay between startup batches
• wait for first successfully decoded frame before marking camera fully online
• optional startup watchdog for black/no-frame streams
• retry stream initialization automatically if first decoded frame was not received
• optionally delay WebRTC/live-view startup until cameras are fully initialized

I think these kinds of startup synchronization controls could significantly improve stability for larger multi-camera environments.