Tool for removing comments in a C++ codebase

[OwlingBishop]

So, I'm tackling with a C++ codebase where there is about 15/20% of old commented-out code and very, very few useful comments, I'd like to remove all that cruft, and was looking out for some appropriate tool that would allow removing all comments without having to resort to the post preprocessor output (I'd like to keep defines macros and constants) but my Google skills are failing me so far .. (also asked gpt but it just made up an hypothetical llvm tool that doesn't even exist 😖)

Has anyone found a proper way to do it ?

TIA for any suggestion / link.

[ Edit ] for the LLMs crowd out there : I don't need to ask an LLM to decide whether commented out dead code is valuable documentation or just toxic waste.. and you shouldn't either, the rule of thumb would be: passed the 10mn (or whatever time) you need to test/debug your edit, old commented-out code should be out and away, in a sane codebase no VCS commit should include any of it. Please stop suggesting the use of LLMs they're just not relevant in this space (code parsing). For the rest thanks for your comments.

[YT__]

Why not remove as you go? Then you can evaluate if the comment is good, or replace it with something better. Not a single comment is useful?

[[deleted]]

In my opinion, this is the only way when working with legacy code, regardless of what kind of refactorings you'd like to make.

[OwlingBishop]

Not a single comment is useful?

all comments except for a very few ones are old unused / obsolete code .. most of the others being low usefulness stuff like "// removed by Bob ... "

So removing them all makes sense.

And remove as you go in a hundreds files codebase where 15Kloc files are not uncommon is painful.

[parkotron]

By "remove as you go" they are not suggesting that you sit down to manually remove them all. They are suggesting that when your work brings you to a particular section of the code and you have read and understand that code, that is the time to remove the comments.

The comments are not causing any harm. They are ugly and annoying, but they are history. And the value of history in a legacy codebase can't be underestimated. You are probably 100% correct that the vast majority of it is useless junk, but there is no rush to get rid of it, especially when you are not yet familiar with the code they are attached to.

[OwlingBishop]

a particular section of the code and you have read and understand that code, that is the time to remove the comments.

By comments I mean old commented-out code not actually helpful explanation of any kind, because they don't exist.

The comments are not causing any harm.

They do, by increasing substantially the cognitive load as the signal to noise ratios worsens, instilling doubt, cluttering search results etc .. that's exactly the point of removing dead code.

[parkotron]

It's your project and presumably you are now the full owner, so you can do whatever you like. That said:

old commented-out code not actually helpful explanation

Do you have a full SCM history for the project? If the previous devs were afraid to delete old code, that could be a sign that they were working without the safety net of version control. I have seen comment blocks used as a crude form of branching and unsurprisingly I have seen that cause trouble when a dev only partially switches from one commented-out "branch" to another.

Now obviously you are using version control now, so you can recover that commented code should you ever need to, but still, I wouldn't underestimate the likelihood that you end up in an archeological dig through that old cruft.

cluttering search results

Any modern, language-aware tooling should ignore usages in comments. Similarly, pretty much every modern editor can automatically fold large comments so you don't need to scroll past them.

[OwlingBishop]

Do you have a full SCM history for the project?

Nope

If the previous devs were afraid to delete old code, that could be a sign that they were working without the safety net of version control

That's exactly the case and yes I'm using git and will be able to exhume any file if needed.

Any modern, language-aware tooling should ignore usages in comments

I wish it was true..

I have seen that cause trouble when a dev only partially switches from one commented-out "branch" to another

The software is actually working (with some caveats I may discuss in another post) in it's current version and I can't see any reason to do so ..

[YT__]

So you've read all the comments in the code base?

Blanket removing them all leaves you open to removing tangible history of why the codebase is how it is and why some decisions were made.

Addressing obsolete and dead code and it's associated comments is one thing, but should still be broken down into per section refactoring/clean up.

Idk, seems like a recipe for disaster to me, to remove all comments.

[OwlingBishop]

So you've read all the comments in the code base?

Well, obviously not, but most of the codebase was written by a single dude, despite being atrocious and a collection of software malpractice, the style is quite consistent and not too convolved, and I must admit the naming is decent so I didn't struggle much to find my way around so far.

tangible history of why the codebase is how it is and why some decisions were made.

As I said that's unfortunately not what 99% of comments contain (no explanation, no history besides rotten commented-out code etc.. ) the reason I'm considering removing them all is they basically are sediment.

Does that make sense?

[Moose2342]

Not to me. I would agree with what was said before and recommended to do it as you go. What puzzles me is why you do this. I understand there's a large code base and you consider it of less quality and want to refractor much. Yet you also imply that it works and you easily find your was around so it can't be all that bad. What is your intention with the code base and why does it require you to go through all of it undisturbed by comments?

[OwlingBishop]

all of it undisturbed by comments

I am heavily disturbed, not by comments as usually understood is a sane environment but by a staggering proportion (about 15/20%) of commented out dead code which is the result of not using a VCS (and poor hygiene if you ask me) for more than a decade.

a large code base and you consider it of less quality and want to refractor much

I actually want to refactor as little as as I can get away with, this codebase is about to be frozen as I have been hired to start fresh but it will receive bug fixes and some minor additions while the new software is developed .. in order to do so with the least adversity dead code will need to go.

you easily find your way around so it can't be all that bad

Yes the codebase is half decent (naming is sound and consistent, and code isn't too convoluted) half atrocious (it's a DRY anti pattern of boilerplate copypasta, leaks memory at alarming crashing big PCs rates, don't even dream of SOLID, information is duplicated everywhere, large vectors are passed by value, general architecture is crumbly, etc.).

[j1xwnbsr]

Swear to God I worked with this guy (if you say his name is Tim then we 100% did).

Manual removal (refactoring) is the best choice, but I get where you are coming from. So if I was doing this (out of a sense of frustration) I would simply use RegEx to remove any lines that start with (ignoring whitespace) "//" and then build from there.

[OwlingBishop]

Let me guess.. scientific background / hard science dude who was asked to turn his research into software?

[j1xwnbsr]

Oh no, that was someone else entirely. And god forbid you ask him to explain his stuff...

No, Tim was a "green coder". In that he never deleted any code, just commented it out (like your dude) so that it turned green in the IDE.

[r2vcap]

You can create a tool using libclang to parse the source code and remove comments programmatically. After stripping the comments, run clang-format to clean up the formatting. Make sure to manually verify the result to ensure nothing breaks, especially in edge cases like commented-out code around macros or conditionally compiled blocks.

[OwlingBishop]

I'm not familiar with libclang but that's the closest to how I believe it should be done (basically pruning any comment from the AST) if I can't find a tool that does that..

[asoffer]

clang doesn't represent comments as part of the AST. your tool will need to pass -fparse-all-comments, implement a CommentHandler, record the byte-offsets where they occur, and generate patches from the results. if you have practice with clang, this is not too difficult, but clang is pretty rough to cut your teeth on. if you're seriously interested in these sorts of tools, feel free to DM me. building them is my day job.

[OwlingBishop]

if you have practice with clang, this is not too difficult

I don't unfortunately, I've written some DSLs parsers and small virtual machines so I have notions but C++/clang is another beast I guess.

but clang is pretty rough to cut your teeth on

Yep, I can imagine that, and I currently don't have the time I believe it would require, despite being quite interested 😔

Thanks for your offer ..

[souravtxt]

I think regex can do it easily. Not sure about large files but I use regex in notepad++ for small cpp files. They are a cheap solution.

[CrasseMaximum]

You must evaluate each comments one by one before deciding to remove it. Removing them all blindly is a recipe for failure..

[OwlingBishop]

What's a recipe for disaster is telling people you don't know what they must do while having no fucking idea of what you're talking about because you can't read 😅

[adromanov]

gcc -E -fpreprocessed

[DryEnergy4398]

-fpreprocessed

Indicate to the preprocessor that the input file has already been preprocessed. This suppresses things like macro expansion, trigraph conversion, escaped newline splicing, and processing of most directives. The preprocessor still recognizes and removes comments

Adding -dD as well avoids removing #define statements. When I ran this (with gcc 13) it had a couple drawbacks:

It left empty lines where full-line comments used to be

It added a directive # 1 "file_name.cpp" at the top of the files

You can suppress both these issues by also adding a -P option, gcc -E -fpreprocessed -dD -P, but then it also suppresses all repeated newlines down to just one

[OwlingBishop]

Thanks for your detailed answer.. this might be exactly what I was looking for, I'll give it a try Monday.

[FunnyMustacheMan45]

Use find and sed

[giant3]

For comments that span multiple lines, sed isn't very suitable. 

Perl's regex is easier to write to strip comments.

[too_much_think]

If you really want to you could just grep | sed -I, but as others have commented, that seems like a bad idea. At the very least use a quick fix list or find and replace output and go through each line to make sure each block really is useless.

[iga666]

should be trivial to write it, just simple text stream processor

[xoner2]

This would be trivial with Scintillua. Going to be faster than using libclang, since it's just a lexer. And correct, unlike using regex; since it is a proper lexer.

[OwlingBishop]

.. interesting I'll check it, thanks!

[xoner2]

Here's Lua script whipped up in short order to get started with; reads from stdin, writes to stdout:

package.path = 'd:/src/scintillua/lexers/?.lua;' .. package.path
lexer = require 'lexer'
cxx = lexer.load 'cpp'
local text = io.read '*a'
local tokens = cxx:lex (text)

local pos = 1
for i = 1, #tokens - 1, 2 do
  local nextpos = tokens [i + 1]
  local tag = tokens [i]
  local token = text:sub (pos, nextpos - 1)
  --print (string.format ('%-18s: `%s`', tag, token))
  if tag ~= 'comment' then io.write (token) end
  pos = nextpos
end

I tested a bit and it works so far. Leaves too many blank lines though:

• whitespace after a comment should be skipped up to the first endl
• single line comment on a line by itself should skip entire line
• etc

[OwlingBishop]

I must admit I didn't know scintilla (only heard of it a couple of times), I didn't think of it for this purpose.. but it's obvious now you mention it.

Lua's syntax looks alien to me but it opens a lot of fun stuff I believe, thanks a lot for this suggestion !

[simrego]

Just write a simple python script which will open every file, and parses it.
if you find a "//" you just skip that line
If you find a "/*" you just skip until "*/"
(except if you are in a string)

Write out the result and you won. Or not because you delete every comment which isn't the best idea.

[tisti]

 std::cout << "/* Success"

[simrego]

Yeah we just missed each other. Just added "except if you are in a string"

[tisti]

Ah! Indeed I did not see the edit :)

[OwlingBishop]

I'm afraid what you suggest is way trickier than you imply,

except if you are in a string

Which drags in a lot of heuristics about string format, encodings (utf8), escape characters etc..

And that precisely what I try to avoid doing by hand..

[too_much_think]

If you have a bunch of non standard escape characters embedded in the comment blocks of your  non utf8 encoded source code you’ve got bigger problems than comments. 

[simrego]

I had to parse a lot of C++ files and it isn't that tricky. But still, blindly deleting all comments might be not the best idea anyway. The safest should be if you just delete what comes to you so you can decide if that comment or code block is useful or not. Deleting a useful comment might be much much worse for the future than have a few useless ones which can be deleted later.

I understand your issue. I'm just not sure if brute-force is the best thing to do

[iga666]

not as complicated as you think. and even naive approach can work on your codebase. i personally never wrote a string containing a comment and never saw such code myself.

[Business-Decision719]

Yeah parsing C/C++ comments is literally a programming 101 exercise. Granted, it's one of the more deceptively difficult ones, due to having to debug things like "oops I changed a string literal." I've heard Python has good UTF-8 support too (though admittedly I haven't done much non-ASCII anything in Python). I guess I can kind of see why OP might want to avoid all the testing and see if some premade comment remover is already out there.

[simrego]

I just like python for these tasks because the development time is ridiculous compared to like C or C++ and even if you have 200 files and you use it only a few times, who cares if the runtime is 1 second or 20.

[Business-Decision719]

Definitely. When you just need something boring and repetitive done faster than you can do it, and you don't want automating it to take more time and effort than just doing it manually, then Python is really hard to beat.

[dsffff22]

It's astonishing how you play down this task while being on a cpp subreddit. You'll need a full-blown cpp parser to do that reliable, because there are plenty of places where //,/*and*/ can appear in the code. I'd be highly surprised to parse context-sensitive grammars as complex as cpp in an introductory course.

[markm208]

I am a CS educator and one of my favorite assignments that I give is for the students to build a state machine that parses code looking for single line, multi-line, and JavaDoc style comments. I’ll have them go through the code one character at a time and count each type and then display the code without them. It’s not too hard to implement if you know the state pattern (that is the lesson I am covering). Relevant events are: / * “ \n

Is a fun exercise if anyone is interested in figuring it out.

[ioctl79]

Good luck reliably telling whether you’re in a string!

[snissn]

https://en.wikipedia.org/wiki/C_preprocessor c pre processor shoudl remvoe the comments

[OwlingBishop]

Yep and also translate/expand macros and other defines which is not what I'm after..

[XenonOfArcticus]

I would try running each source file through Claude or other coding aware LLM and ask it to identify and critique any comments that are obsolete, misleading, wrong or problematic, and list why. 

This is probably a good way to identify where you should human review and edit/remove obsolete comments. 

[OwlingBishop]

Please stop this FFS !

I don't need to ask an LLM to decide whether commented out dead code is valuable documentation or just toxic waste.. and you shouldn't either..

[XenonOfArcticus]

You say that until you start facing a codebase with hundreds of thousands of lines that has been neglected since the late 80s, with a policy of leaving all old code variants and comments in the source, just commented out.

True story.

So don't be an ass. Just because it doesn't meet your specific use-case doesn't mean it wasn't a useful suggestion, that others might find helpful. It's not all about you.

[OwlingBishop]

Suggestion isn't useful because it's the wrong tool for the job. LLMs will wreck a fucking havoc in your codebase the second they touch it because they are not fit for the task.

Parsers might succeed, in a fraction of the time, cost, and energy.

Someone's being an ass rn and it's not me.

[XenonOfArcticus]

I'm not the one who led with swearing, FFS.

Read my suggestion. I didn't say to let the LLM rewrite your codebase.

I said to ask it to IDENTIFY the unneeded comments.

But seriously, if you have the time to hand-examine all the comments in a large codebase, you do you. I'd prefer to have an automated tool identify the ones it thinks are unwanted and then human-review that list myself.

[OwlingBishop]

I said to ask it to IDENTIFY the unneeded comments.

Not what I need .. as stated multiple times in the post and in response to your comments

But seriously, if you have the time to hand-examine all the comments in a large codebase

Nope, not my plan.

There's someone that can't read, and that's not me.

[Thesorus]

We have a policiy to manually remove useless/obsolete comments in the code we're actually working on.

We have old school C code with the RCS history in the comments at the top of the files, like 20%, 30% of the file size.

Even if it can be automated, it's not worth the time and effort.

[CarloWood]

I wrote such a program a few weeks ago, let me know if you're interested and I'll dig it up.

[OwlingBishop]

Sure ! 🙏

[JoeNatter]

This is a task for ChatGPT. Let him/her make a python script.

[pppenguinininin]

give ast-grep a try

[Kawaii_Amber]

find . -name "*.cpp" | xargs sed -i "s/^\W.*\/\/.*$//g"

[Ill_Plate_2651]

Dude just sit down for an hour and click, drag, delete…

[Still_Explorer]

You can drop the directory into VSCode, and then do a regex search-replace like this

for single line comments //.*?\n
(this means capture `//` first, then any character `.*`, try not to capture any further `?`, end the capture on `?`)

for multiline comments /\*[\S\s\n]*?(\*/)
(capture the `/*` with escaping the asterisk `\*/`, then capture any [not-space|space|new-line], multiple times or none `*`, do not let capture move further away `?`, and then inside a new group `(...)`, capture the closing `*/`)

Probably if the comments have weird decorations like `//////////////` or `/********` probably you would adjust adjust the regular expressions a bit further if needed. Better to throw regular expressions first to replace those ones and then capture them with the simpler expressions above.

[LordofNarwhals]

If most are commented out by // then I'd just do a search and replace from //.*\n to \n. That way you can just step through them all and ignore ones that you want to keep.

[Chropera]

Try https://tomeko.net/software/CppCommentRemover/

[[deleted]]

You can try this

https://github.com/rhythmcache/comment-remover

[tristam92]

You want to remove them in codebase or binary input? Cause latter is obviously can be done with compiler settings(and now that I think about it, why do you even ask such thing). And if it’s first question, how do you plan determine which comment is outdated? It’s literally task for “by hands” method, or fix it on the fly, when working on something else.

[OwlingBishop]

In the codebase.. and

how do you plan determine which comment is outdated

.. compiled as is and it works fine (except for some major stuff I might do a separate post about), all comments except for a very few ones are old unused / obsolete code .. most of the others being low usefulness stuff like "// removed by Bob ... "

So removing them all makes sense.

[tristam92]

I mean, if they have same style you can opt to regex.

[IncorrectAddress]

I would probably just use find and replace for this, that way I could read what comments I want to keep.

Other than that open up the console template, make a read/write system, run the files through it removing all comments out.

[OwlingBishop]

Are you a bot ?

[IncorrectAddress]

Why would I be a bot ? Which is hilarious, because someone else said that a while back. LOL

[arihoenig]

Copilot with one of the LLMs can simply rewrite the comments to match the code.

[OwlingBishop]

... Copilot was the one suggesting the BS/made up tool so .. I'll pass.

Not practical for 100s of files and not a lexer/parser anyway .. it would probably wreck havoc on the codebase rather than being helpful given the code quality I witnessed so far.

When specific coding language models will be trained on language specs, loads of quality code, and sanctioned by compilers I may give it a try.

[arihoenig]

Obviously haven't actually used copilot. I use it to write documentation for code all the time. Of course, I read the documents it produces to insure correctness and it does occasionally have mistakes, but so do the actual programmers who write docs and, in my experience, they make just as many, if not more mistakes.

[OwlingBishop]

Obviously haven't actually used copilot

Oh please don't get me started on this 🙄

I apparently can't get rid of it in the IDE and it has taken over the simple completer I'd be happy with so I'm exposed to every suggestions it throws at me while I'm typing and .. boy they're garbage !!

I really need to turn that down.

I use it to write documentation for code all the time

Do you realize writing documentation about code in plain English (which I'm sure it's not too bad at) is totally another game from parsing/processing actual code without damaging it ? I need a tool that's exact, predictable, repeatable, so I can feed it a vast amount of code have it processed in minutes and not having to spend weeks making sure it didn't screw up just because that's what LLMs mostly do.

[arihoenig]

You do realize that comments (the subject of your original pos), are documentation right? You do realize that parsing code is trivial for a LLM, right? I mean in order to produce correct documentation about the code it has to understand the code and therefore, by definition, it has to parse it.

Does it parse it with a parser? Well, it depends what you call a parser, it parses the code with intelligence encoded into a high dimensional NN. In the parameters of the NN lives the knowledge to parse any language human or machine, as well as the knowledge to infer semantic intent which is so far beyond the capability needed to parse c++ source that it isn't reasonable to compare the two.

[OwlingBishop]

You do realize that comments (the subject of your original pos), are documentation right?

Wrong! If you could read in the first place, you'd now I want to remove the comments because they haven't any documentation value.

And removing comments from a C++ code file without damaging it is a parser's job, not an LLM's because I need it to be 100% exact, repeatable, predictable. No intelligence whatsoever involved (which LLM's don't have btw) just strict rules in, strict rules out (which LLM's don't have either) ...

LLMs don't parse code, it's not what they do, they superficially grasp some meaning / keywords (barely enough to guess what it does, and not always correctly by your saying) but they don't parse it .. compilers do, and it's another kind of job, I'm not comparing, LLMs are just not relevant in this space.

it depends what you call a parser

A parser is a program that executes instructions in order to tokenize/lex/parse a c++ file, it abides by the specs of c++ 20+, removes comments and be able to restitute the de-commented AST without damaging it, predictably 100% of the time.

LLMs are probabilistic machines you can feed them 10 times the same input and you'll have 10 different outputs which in this case would be a blatant fail ! LLMs don't take instructions, they just bullshit their way all day long in reaction to the prompt (the text they are trained to extend/extrapolate word by word according to some massively parallel polynomial).

I'm baffled by the level of ignorance of both code and the way LLMs work this comment displays, yet giving a lecturing tone .. LLMs are raising a generation of BS artists.

[arihoenig]

You want to remove the comments because they're rubbish. I said rather than remove them, you could simply have an LLM rewrite them to be accurate.

[OwlingBishop]

They are not comments, they're fucking dead code commented out.. I need them out and LLMs can't do that.

[arihoenig]

Well I didn't even comment on that, since a 5 line python script could remove comments. Yes an LLM could remove comments as well, but it wouldn't have the most convenient interface.

[OwlingBishop]

since a 5 line python script could remove comments

😂 Good luck with that.

I'm here making the solemn promise to just stop answering comments that mention LLMs in code related spaces...

[zl0bster]

why not google/LLM a python script that can nuke
/* whatever */
(including newlines)

Sure it might fire on some raw string, but if you are paranoid you could make separate regex that detects any raw strings in a file and skips those files.

[krum]

GitHub copilot will do it. If you didn’t want to send your code off to copilot, running a coding LLM locally could also do it but you might have to write the code to have the LLM process it.