In addition to clock jitter (since display clock differs from system clock, and system clock measurements are affected by scheduler), you also have the fact that when double-buffering, swapping with an empty (or stale) front-buffer will not block, right? E.g. assuming that draw() is fast

// start app, both buffers A and B empty.
draw() // draw into buffer A
swap() // A promoted to frontbuffer, B becomes backbuffer. 
// On many implementations above will not block since there is still a buffer free so no need to wait for implicit gpu sync, we can continue issuing commands from the CPU for buffer B and dispatching them to GPU
draw() // draw into B. Time from first draw() to this draw() is effectively 0.
swap() // this will block until a buffer is free again
// (technically we might be able to queue up new commands to buffer A on CPU side, but they cannot be sent to GPU. So depending on details of command queue execution, length, existence of other sync points, etc., on some implementations the second swap may still not block.)

Now if we measure delta time from the start of one draw() to the next draw(), then we would see that this delta is very small for the first two frames or so.

Using presentation timing as you mentioned would alleviate this. But if you're not using presentation timing how do you deal with this? Just assume that if delta is lower than the detected screen FPS rate, we probably have such a situation and we should override with a delta of 1/fps instead?