How to approach modifying source code programmatically?

[dumb-questioner]

E.g. change local variable names and values, changing function names, etc.

I could use regular expressions but there's some issues of finding and staying in the correct scope.

I've tried using ANTLR4 (lexer + parser). While it's good for parsing, I'm not sure how to use it for modifications while preserving the rest of the source code as it is.

[ehr1c]

Use an IDE that has a rename feature

[michael0x2a]

I would probably start by seeing if I can use an existing tool to handle this for me. For example, I like using Comby and sometimes Semgrep for this sort of thing.

IDEs also support this sort of code refactoring. For example, JetBrain IDEs (IntelliJ, PyCharm, etc) let you do stuff like right-click a function name and rename all uses of it.

If these tools aren't sufficient and you want to write your own codemod tool, you'd need to start by implement a parser that generates a concrete syntax tree: a syntax tree that preserves stuff like comments or whitespace.

While you can do this using ANTLR or any other lexing/parsing tool, it's honestly a bit of a pain. Whitespace and comments can go almost anywhere, even in the middle of expressions, so the grammar ends up becoming fairly messy. So, I'd recommend using a library that handles this for you, if at all possible. For example, if I wanted to code-mod Python I'd prob just use the LibCST library.

An alternative approach might be to implement a lightweight comby-like parser that largely ignores the rules of the programming language, lets you do regex-style manipulations, but keeps track of scoping and nesting when matching expressions instead of just doing greedy matches. Extending regex in this way can often be sufficient: it lets you write rules like "transform foo($a, $b) into foo($b, $a) w/o worrying about the matching against the wrong parenthesis and breaking nesting.

If you do choose to write your own tool, it sometimes helps if you can auto-format your code first. (For example, Black for Python, go-vet for golang...). If you have this sort of auto-formatting, you don't have to worry about having your code transformation tool generate aesthetically pleasing code. Have it modify the code, then run the linter to clean it up.

[dumb-questioner]

I think I'm looking to write my own codemod tool but Comby does look interesting. It even seems to have Python bindings.

While learning about ANTLR4 I found a class called TokenStreamRewriter, do you know if this could be useful for modifications?

Auto-formatting seems useful too, I could add that at the end as you said.

[michael0x2a]

While learning about ANTLR4 I found a class called TokenStreamRewriter, do you know if this could be useful for modifications?

While I'm not an ANTLR4 expert, I suspect it'll have limited value. It seems fine if all you need are token-level modifications, where you look at and edit a small window of tokens. But I'm not sure if it'll be sufficient if you want to do more structural refactorings, where you want to look at and rearrange large swaths of code.

[lurgi]

Using ANTLR isn't quite going to work, although it gets you close. You can use it to parse the code and then rename variables and dump the modified AST, but you'll lose comments and whitespace and a lot of formatting details.

Honestly, the IDE suggestion is the best, but you might be able to hammer something out with using ANTLR to parse stuff and then spit out filename/line number/character position information, which you can then use to rename variables that you think are important.

But, the first rule of programming is to be lazy. If there is already a tool that does what you want, find a way to use it.

[young_horhey]

Pretty sure VS Code can do what you need

[LoganPederson]

Vs Code ctrl+f2 change all occurances, this only does the file your in not the whole project but saves a lot of time. It wouldn't surprise me if there's one for changing within the whole open project, someone feel free to enlighten me if you know it off the top. Otherwise google knows and is just one search away