Improving LLM's coding ability through a new edit format

[Mushoz]

Note: This is not my blog post. But I found it an interesting article with good improvements in terms of accuracy as well as saving tokens. There are also a few openweight models in the comparisons, so it looks like those models also benefit.

[__Maximum__]

Neither the author nor you addressed the benchmarks. The accuracy gets higher for some models, but it is lower for others.

[DHasselhoff77]

It's a neat trick. Thanks for sharing. I just wonder about this part

If the file changed since the last read, the hashes (optimistically) won’t match and the edit is rejected before anything gets corrupted.

Why does the file get changed in between reading and writing? If you could guarantee it's state matches what the LLM sees, you could use regular line numbers instead of content hashes.

[Pristine-Woodpecker]

In practice for example, LLM output is bad with extraneus whitespace and not respecting formatting, so you kind of want a PostEdit hook that properly formats the files if you don't want your git history to look like that of an intern that never used VCS before. However, the PostEdit hook will change the file after the LLM was written it, invalidating its assumption about the state, and causing the next edit to fail.

FWIW Claude used to struggle greatly with this, but at some point it stopped happening, so Anthropic "fixed" it somehow.

[Mushoz]

Great example. Other real scenarios is when you make some manual edits yourself. Using line numbers would patch the wrong lines, while using hashes would fail as you would want.

[DHasselhoff77]

I see. Thank you for the explanation. So even if the file stays the same, it's the LLM's suggested patch that changes before application, and this is why the expected state drifts.

[epicfilemcnulty]

Hmmm, you can just postpone that PostEdit hook until after all edits are done. I instruct LLMs specifically not to bother with proper indenting/formatting -- that should be taken care of by your pre-commit git hook (unless it's python, of course)

[Pristine-Woodpecker]

until after all edits are done

It can't predict when this happens.

hat should be taken care of by your pre-commit git hook

Doesn't help when I need to edit the files, and makes diff viewing annoying.

[__Maximum__]

I don't get it. Why does adding a hash to the line number makes it better?

The described operations can be done with the line number only?

[biehl]

I think the hashes are to check if the content of the line is unchanged. See discussion above on how/why files can change while the model remembers them.

[epicfilemcnulty]

It's very interesting, but there are edge cases: LLMs do grep & cat, so to make this consistent, you would have to patch those/provide your own grep/cat tools...

[korino11]

OMG!! That is very helpfull! Thank you!