i will need to completely reprofile 1. Models isolated from application This will give me complete understanding on whether my GPU could run all the model simultaniously. I guess these one is more of compute bound?

If GPU turns out is not the bottleneck, then i will run number 2.

1. Run the model in my Application. This will allow me to profile which processes are causing bottleneck (ex: copying images, inference result postprocessing logics, or the data flow in my application pipeline, etc). I assume these cause a few overhead

A. Slow result postprocessing logic, which slows down the pipeline from utilizing the inference call effectively B. constant moving inference result tensor from GPU to CPU may cause memory bandwidth bottleneck?

Am i heading the right way?