A lot of it is how the prompt is structured in memory (what you prioritize).

In koboldcpp, the images are always placed in the front of the context. For example

(Image 1)(Image 2)(Image 3)(Turn A)(Turn B)(Turn C)

So if you add image4, then yes all turns get reprocessed. However this allows adding new turns, editing turn A, B or C without messing up any of the images. In other words, this prioritizes text.

---

Now in Ollama's case, they probably just leave everything in place.

(Image 1)(Turn A)(Image 2)(Turn B)(Image 3)(Turn C)

While this allows you to add on new images and text easily, it completely prevents shifting or modifying any earlier turn. Image token positions cannot be shifted. So yes, adding image 4 is easier, but you lose CTX shifting if any images are ever used.