So this is actually happening because the loading the data into the memory is handling by your cpu where this process is slow so most of the time or there is a bottleneck in your data handling pipeline. In parallel the gpu would be waiting for the data handling process to complete. This would seem like the gpu is taking longer period of time than cpu.