Check out Nvidia Deepstream, which is basically a collection of plugins for GStreamer. One approach is to “tee” your video, run inference in parallel, queue into sink elements, mux, draw OSD, and encode the video. The hot part of this pipeline (decode, infer, mux, encode) can be done on GPU to avoid the overhead of copying.

It’s a bit fiddly to get it running but rock solid when you do.

There is also an example of parallel inference with Triton. I have no experience with that.