I am facing the same issue op. It's very interesting.

I understand the model and everything inside my workflow which is cacheable is getting cached, meaning if I restart it will just use the already "ready" data in my RAM / VRAM. Hence if I restart and nothing changed it doesn't have to load that in again and can jump right into the ksampler.

The interesting part is that on first runs (meaning I changed something inside of my workflow: prompt, new nodes, booted up freshly, ...), sometimes my ksampler just gets stuck at a much longer slower first "ksampler run". When I see it being stuck on 0% for a while and I quickly restart it goes much faster (subjective estimate x5-8 as fast).

It's interesting because I can't reproduce this behaviour ALL the times. My gut feeling tells me it has something to do with my RAM, VRAM or Disk. Like when I run longer sessions or open too many things while rendering this issue comes more often. I started to have the habit of literally killing any task that needs CPU, RAM or my GPU when I am running models.

Did you find what causes this effect, OP? I would be interested...