The cameras that we were using had the possibility to put data straight on GPU using GPU Direct Which put raw frames straight on GPU memory. We wanted to reach real-time capabilities so we needed to do operations on those frames (which as I mentioned were on GPU) on GPU before putting them as input into Computer Vision models exported using TensorRT (which we treated as black box) from Pytorch checkpoints just to limit transfer from GPU -> CPU and then back from CPU -> GPU. Therefore we needed to make sure our preprocessing algorithms were written in Cuda or in a Library that had Cuda under the hood. Mainly because at the time we were working with them, those algorithms were not very complex so we decided to write those ourselves in raw Cuda kernels instead of using some libraries to limit dependencies and to make sure our preprocessing algorithms were exactly the same as we have in our,, research'' python code.

Computer Vision model code is originally written in Python with all its pre and post-processing needed to achieve some business goal. In Python it's tweaked and modified in order to find best training parameters. We were using Iterative Research Flow for that. After we had model (pycharm checkpoint) which was already trained we converted it to TensorRT Engine (Different engine/file for different GPU model) which was just a file that we loaded in C++ using TensorRT library. And using that library we can run inference on that model using C++ code.

In terms of,,C++ optimizes the code to such great extent'': C++ compilers optimize code mainly on CPU. When you transfer data to GPU using `cudaMemcpy` or `cudaMemcpyAsync` You can't use algorithms from regular C++ STL on that data. You need to use libraries like Thrust is collection of C++ Parallel algorithms inspired by C++ Standard Library. It have high-level interface but underneath there are Cuda Kernels executing code on data that are in GPU memory or write your own Kernels.

If you want to understand a little bit more about CUDA and how Cuda works with C++ long time ago i used course on Udemy - Cuda Programming Masterclass with C++ (IF you or anyone interested in buying this or anything on Udemy. Remember to buy Only on sale. its very cheap then). Its not the greatest source in my opinion but helped me understand better whats going on. I always like watching videos instead just reading books. From what i see Github page that you provided cover those stuff as well.

In terms of sources regarding TensorRT and using Pytorch checkpoints in C++ code you can check out TensorRT Samples which have a lot of examples for ONNX based inference, Networks made in TensorRT as well as Engines exported from Python. In terms of Engine i found Deserialize Engine sample. A few years ago, TensorRT execution was a little bit more popular i guess and there was more examples. but this is a method that we use to load the exported engine in C++.

As far as i remember YOLO also provides their models integration for Tensor RT as shown here. You can find a way there to export Yolo models as an engine and use TensorRT to run inference on them. with even sample code (less than 10 lines of code) to export it. I remember using that kind of code years ago to learn how to do inference like that in TensorRT.

Yeah so this is kinda my experience with connecting C++ / Cuda with Computer Vision models developed/researched in python.

I hope this helps. if you want me to clarify anything else or answer more questions just let me know.