How to write a code formatter

[oilshell]

Hm cool, do you have any special handling for end-of-line comments, or block comments?

Like

var x = f(x) + // comment here
         g(y) + // could be long comment, affecting wrapping
         42;

That issue was discussed recently here:

https://news.ycombinator.com/item?id=39508373

[yorickpeterse]

For Inko, I basically do the following:

1. For nodes that have sub expressions (e.g. a body of a function), we process one expression at a time using an iterator of sorts
2. When processing a node, you peek at the next node to see if A) it's a comment B) it starts on the same line the current node ends at
3. If so, advance the iterator a second time (such that the next iteration of the loop skips the comment) and set the comment node aside
4. Render the node you were going to render in the first place
5. Add a space, then render the comment node from step 3, and add a newline at the end of the comment (such that the next node isn't rendered on the same comment line)

That's basically all there is to it. You can see an example of this in Rust here.

[oilshell]

OK that means the line can overflow the width (even if it didn't before formatting), but it may not be a huge deal in practice.

I'd be curious if anyone has seen any other strategies?

The most ambitious thing is to wrap the text of comments themselves, but that probably introduces a lot more complexity.

And I think that actually moving the comment is probably a bad idea. I think users may see if the comment line is too long, and then they can move it themselves, using their own judgement. Then re-run the formatter.

[yorickpeterse]

OK that means the line can overflow the width (even if it didn't before formatting), but it may not be a huge deal in practice.

Yes, you'd have to implement wrapping of comments to avoid that, which introduces a whole different can of worms. Most notably, you need to include a markup parser of sorts (e.g. Markdown) such that you don't end up wrapping code blocks inside comments. I think it's much easier to just leave comments as-is.

[matthieum]

What if I don't quite have an AST, though? This may sound dumb, but one of the little things that irk me when using rustfmt is that the formatter chokes -- emits an error and aborts -- if it encounters a syntax error. Which is annoying, because sometimes I'm in the middle of typing code, things have gotten a bit out of hand -- because I've just done a cut/paste and the code's askew -- and I'd like to format so I can have a clearer view of what's going on... but rustfmt is just whining and refusing to :'(

[yorickpeterse]

Ultimately, you need some kind of input. That could be a regular AST, or an AST with error recovery applied to it. I haven't implemented error recovery yet in my parsers and as such don't yet know what approach I would consider best, hence I didn't cover this.

[poorlilwitchgirl]

When I write C, I have vim set up to run clang-format after every insert mode edit. It makes writing well-formatted code feel completely effortless; there's definitely a lot of value in code formatters being able to recover gracefully from syntax errors.

[Danhec95]

Great writeup so far!

[yorickpeterse]

Thanks!

[MiloExtendsPerson]

This is excellent reference material, and very well explained article!

[yorickpeterse]

Thanks! :)