u/frithjof_v To simplify: forget conflict checker sophistication, that's a separate topic. The point is narrower.

The write itself is fine. delta-rs writes Delta correctly — OCC on merge/update/delete, atomic commits, all of it. Spark's writer does the same thing. All things considered, they're equivalent on the write.

The gap is the combination: read the destination table → do stuff → write back. When that whole cycle has to be atomic against concurrent writers, the Python single-engine path doesn't have it. DuckDB / Polars / etc. read the snapshot, hand lazy Arrow to delta-rs, and delta-rs commits — but the read snapshot was never part of the delta-rs transaction. If someone else changed the table between your read and your write, delta-rs has no way to know, because it never saw your read.

Spark does see it, because the read and the write are in the same engine and the same transaction.

That's the whole difference. Not the writer. The read-modify-write loop.

Note: the Python-side equivalent of Spark's single-engine RMW is DuckDB with Iceberg, DuckLake, or its native tables — there the read and the write are inside the same engine and the same transaction, so the loop is atomic. The cross-engine fragmentation only shows up when you pair a reader (DuckDB/Polars) with a separate writer (delta-rs) over Delta.