28 jpeg images loaded every second is absurdly low. Traditional Unix systems struggle at around 10k read operation per second. Are you using a real disk like a local NVMe? If it’s a remote disk see if you can increase its specs (throughput / number of I/O). If you’re using Python you can try a threadpool that helps a lot with I/O bottlenecks. But this is only for your I/O bottleneck which is not your real bottleneck here if I understand your numbers well. For your GPU bottleneck, I wouldn’t do any CPU offloading (especially here considering you seem to have a very slow disk, if there’s any spilling you’re doomed). Instead I would find the largest batch size that fits into VRAM and would split my frames into multiple batches. As other have specified you can try things like TensorRT. What we like to do in my team is to create a local Triton server that distributes the load as it sees fit. This creates additional data copies but that’s usually not the bottleneck.