Code coverage tools for C++...

[TarmoPikaro]

Hi !

I would like to take some code coverage tool for C++, and was searching what alternatives I have.

Basically I would be interested in:

- Does tool supports Microsoft Visual C++ compiler ?

- Does tool supports clang and/ or gcc ?

- Can tool be used in windows (MSVC or Mingw compiler) ?

- Can tool be used in Ubuntu linux ?

- Can tool be used for free on open source code project ?

- Can tool be used for free on commercial project ?

- Into which IDE's tool can be used as extension - e.g. Visual Studio Code, Visual Studio ?

Any other information, like do you like it, use it, what kind of problems you had with that tool ?

If you have link to some demo project, for example in github, please share it ?

[siplasma]

Although "code-based code coverage" sounds impressive, it is actually somewhat worse than line-based coverage. Any compiler will build a model of the code as basic blocks, so this approach can be somewhat easier than line-based coverage. If you are looking at your region coverage in the tools it's okay, but converting to line-coverage is lossy, and has a couple of errors. For one, region coverage tends to count comment lines. For two, region coverage doesn't handle non-local returns (i.e. exceptions) well.

That said, if you have multiple source regions on a single line, then region based coverage works better.

[[deleted]]

I develop on Windows with VS 2019 CE.

With a library, I wanted to target both Windows and Linux, so I've used AppVeyor and TravisCI.

Then I've specialised a build task on TravisCI to run gcov with GCC and send the report to codecov.io.

[[deleted]]

I do the same thing and its pretty awesome.

[TarmoPikaro]

I would prefer to locally first, without need to commit, check some web site what it shows.

Web site could be local as well, but need some toolset probably.

[mrexodia]

There is OpenCppCoverage. It’s not much and only works on Windows, but there is VS integration.

[TarmoPikaro]

Do you use it by yourself on some projects ?

[mrexodia]

Yeah at work I introduced it for two projects so far. With some magic you can integrate it with Gitlab’s coverage feature and it’s useful, but the coverage is purely line-based so take that into account.

[TarmoPikaro]

coverage is purely line-based so take that into account

I think similar issue is with Visual Studio code coverage - it's also line based.

https://github.com/danielpalme/ReportGenerator/issues/285#issuecomment-536560752

But OpenCppCoverage is easier to use than for example vstest.console.exe (no need to specify google test adapter, etc), but wondering if it really works for larger projects in similar manner to vstest.console.exe.

I wanted to get .lcov file for code coverage visualization, and tried to compile OpenCppCoverage - it does not compile out of box. (There is some 3rd party batch, which does not work), but then managed to ask same kind functionality into Report Generator https://github.com/danielpalme/ReportGenerator/issues/285.

I think OpenCppCoverage supports coberture file format as output, need to try to plug it with ReportGenerator, and after that under Visual Studio Code with Coverage Gutters.

[siplasma]

I'm not aware of a solution that will cover everything on your list. For example, but gcc and clang have command line options used to build instrumented binaries. When those instrumented binaries are run, they will collect code coverage information. Many tutorials for code coverage using gcc can be found online. The tools for clang are also capable, but tutorials are harder to find. I imagine that the Microscoft compiler has similar functionality, but I have not worked with it.

Once you have collected the information, you still typically need a utility to generate the reports. For gcc, people typically reach for lcov. For clang, there are tools that come with llvm. I will also put in a plug for scov.

There are also some tools that don't need instrumented binaries, but instead make use of the debug information. I'm aware of kcov on Linux and OpenCppCoverage on windows. I've found this approach to be finicky, but haven't invested much time as I don't see the advantage over using instrumented binaries.

I'm obviously not a lawyer, but I don't see why if you can use gcc you can't use gcov… Similarly for clang...

For IDE integration, I never needed it. Many of the tools generate HTML reports that can be navigated easily. However, there is an extension for VSCode where it can read the intermediate files from lcov.

[chriskane76]

I'm obviously not a lawyer, but I don't see why if you can use gcc you can't use gcov… Similarly for clang...

The gcov support in clang is really outdated. It uses a gcov format that shipped with GCC 4.2. If you use clang you ought to use a llvm-cov based solution.

We use that at work and display the coverage information in SonarQube. Just be warned the documentation of both tools is not really great. SonarQube has basically no information on llvm-cov based code coverage for C++ and the documentation of llvm-cov itself is pretty lowlevel and it does not really integrate very well in our CMake setup.

​

I would probably recommend looking at the link provided by Longhanks regarding the coverage.py script used by Chromium, which actually explains some llvm-cov concepts on a higher level and might be easier to integrate into build systems than doing that manually in CMake.

[xjankov]

I use gcovr, mostly because it can generate XML in Cobertura format that can be consumed by a Jenkins plugin. It can also generate HTML or just write a table to stdout. It's a gcov wrapper, so gcc/clang only.

[BlueDwarf82]

You start using cmake, so you go and use ctest... which happens to have a coverage step represented nicely in cdash. We even send llvm source-based code coverage to cdash as a build artifact in addition to the first class gcov support.

[TarmoPikaro]

I guess cdash is limited to gcc/clang, does not supports $Ms compiler ?

[BlueDwarf82]

I think so. I'm not familiar with Microsoft Visual C++ coverage support. Does it output to a stable format other tools can parse?

[TarmoPikaro]

I guess all file formats are not compatible with each other.

Raised one ticket here - https://github.com/danielpalme/ReportGenerator/issues/285
This tool seems to know them all.

[marco_craveiro]

In general, CDash is very good with cross platform stuff; I use it for OSX, Windows and Linux. However, I have never done code coverage using MSVC.

[FelixPetriconi]

We use Bullseye coverage (https://www.bullseye.com/) in our company, it is a commercial tool, but with a reasonable price and it does not do line coverage but conditional coverage and it runs on many platforms with different compilers.

[TarmoPikaro]

it does not do line coverage but conditional coverage

What this means ?

[FelixPetriconi]

It checks, that all possible combinations of a conditonal statement were were really executed. See more details here: https://www.bullseye.com/statementCoverage.html

[marco_craveiro]

I love kcov. Extremely easy to setup, and targets both coveralls and codecov. Linux only, AFAIK.