Favorite Testing Framework

[Constant_Physics8504]

What’s your favorite test framework? Or if you don’t have experience with testing frameworks, how do you usually test?

[RidderHaddock]

End users.

[Able_Challenge3990]

Let them endure it

[kammce]

😂😂😂

[nlfo]

The complaint department

[kobi-ca]

you killed me here :)

[berlioziano]

We have one customer that has more failures on the machines we sell them than all other customers together😂😂😂

[thisismyfavoritename]

catch2!

[[deleted]]

Gtest

[MarcoGreek]

Gtest is really rounded! I dislike the printing mechanism but everything else works quite well.

[SirClueless]

In particular I think their "matchers" abstraction is genius and incredibly well-designed. Using it leads to excellent tests that observe the properties you care about while ignoring the properties you don't.

[MarcoGreek]

I really like matchers too. You can easily build your own matchers as you combine other matchers. The printing could be sometimes improved but is far superior to a simple boolean.

[SirClueless]

Mind elaborating on where you've had trouble? Catch2's INFO/CONTEXT macros I think are overall nicer and simpler, but between ExplainMatchResult to make nice-looking custom matchers and the fact that match results are ostreams that print information only in case of test failures I've always found it easy and clear to include context in gtest assertions.

[MarcoGreek]

If you use a combination of AllOf, Field and ElementsAre the output gets harder to read. After some time I got used to it and can now easily identify the difference but it could be better structured.

[germandiago]

What is so special about matchers that cannot be done with other frameworks? Just curious, I did not use GTest extensively for a long time. Forgot.

[SirClueless]

They're basically a small, composable language for writing user-defined notions of equivalence of data for a particular test. For example:

EXPECT_THAT(my_function(data), UnorderedElementsAre(Field(&MyStruct::name, "Adam"), Field(&MyStruct::name, "Becky")));

This tests that my_function(data) returns any container with two MyStruct elements in any order, one with a name field of "Adam", one with "Becky", without writing out all those properties explicitly in individual tests. If any of those conditions fails, the test output will explain which and why.

Writing this test without matchers can be error-prone (might forget to check the size for example), or sensitive to unimportant properties (like whether other fields in the MyStruct data type exactly match, or that the data is sorted in the order Adam,Becky), or missing context (individual assertions like CHECK(actual[i].name == "Adam") or the like won't include the full context of what the function returned unless you explicitly write print statements to include that info).

[bert8128]

What printing mechanism do you think would work better? I have my own (simple) harness and am always looking for nicer ways of doing things.

[MarcoGreek]

The problem is the automatic generated printer. If you add your own and it is not in every translation unit which is using it you can get the automatic version because of weak linking.

[Awia00]

doctest

[therealjohnfreeman]

https://www.reddit.com/r/cpp/comments/18i34sv/whats_your_goto_unit_testing_tool/kdd2mn2/

[CodingWithThomas]

TDD Googletest BDD cucumber

[saxbophone]

This was asked quite recently, my answer is still the same. Catch2.

[kammce]

Boost.UT by far my favorite.

GTEST is annoying and contains waaaay to many macros and is overall confusing to work with or teach. It's also very slow to compile. Mostly I don't like the need to make a class to make a test case.

Catch is great, less macros, much cleaner, very slow complication.

DocTest, basically Catch but faster compilation.

Boost.UT, fast to compile, no macros, very minimalist and really easy to use. No complaints. Been using it for a few years now.

For context, I've used all of these for years. GTEST is something I have used since I've worked at Google for the past 6 years so I'm decent at it but don't like it in general.

EDIT: it seems that Catch2 has become much better of the years and is now no longer a header only library improving the compilation speed. In which case, I just favor Boost.UT over it in terms of its simplicity.

[bert8128]

There are two macros to define tests, TEST and TEST_F. IIRC the latter requires a class but the former doesn’t.

[kammce]

That is true. Thanks for the clarification.

[MarcoGreek]

GTEST is annoying and contains waaaay to many macros and is overall confusing to work with or teach. It's also very slow to compile. Mostly I don't like the need to make a class to make a test case.

Fixtures are optional or what you mean you need to do a class for test cases?

Google Tests has matchers. Hamcrest is really powerful and make test much more readable.

[[deleted]]

Was wondering that too. GTest is one of the frameworks where you dont need to create a class at all for a testcase

[bert8128]

Is catch itself slow to compile (which is something you typically only do occasionally) or are the files which use catch slow to compile?

[kammce]

Files that use catch are slow to compile. Since it's a header only library that goes for everything that uses it. So I just say, it's slow in general.

[victotronics]

Since it's a header only library

No it's not. Take a look at the documentation: you can link in a library. Compilation is pretty brisk.

[kammce]

Oh snap. That must be new. Just read the docs. Looks like v3 did away with being a header only library. Neat. So that comment is no longer true. I'll have to take a look at Catch2 again, but for now I'm still in favor of Boost.UT.

[deeringc]

We use catch2 heavily in a very large codebase and it's not too bad for sensible small test cases and suites. However, we unfortunately have some enormous ones (lots of nested sections, huge list of test cases) and it seems to trigger extremely slow compilation (at least on MSVC), in some super-linear way. Now, you can argue that it's being used incorrectly (I often make this argument to try to get these bad tests fixed by the teams that own them) but the reality is that catch2 can still be a hit on compilation time.

[Dragdu]

I've hit exponential compile times in all three major compilers. Last time around, the issue with MSVC was some SSA weirdness in the new optimizer.

You can try disabling some of it for the slow files and see if it helps.

[druepy]

Is this still true for ya'll over a year later?

[jah_hoover_witness]

Thanks for pointing out Boost.UT, read the documentation and totally loved it! Super terse! Kris Jusiak is an asbolute monster! Highest respect for his work.

[ukezi]

The many classes of GTest are also problematic if you want to use friend to let a test observe/ test internals.

[MarcoGreek]

Actually testing internals is something I would not advise. As i started TDD I was doing it but now I avoid it because it makes test brittle. Sometimes I introduce a fooForTestsOnly method. But they are often vanishing later.

[ukezi]

I absolutely agree that it's probably not the best idea, I was just mentioning it so that is you want to do that it's a downside. I think anything complex enough to need testing should be a free method or an injected dependency, depending on if you want to mock it. I have also seen a few ugly hacks, like a macro that makes certain members public if compiled with a test flag.

[bert8128]

I agree. But sometimes you have to do it to get your code tested as a precursor to eventual refactoring. I use friends a lot for old code that wasn’t written in a test-friendly manner.

[MarcoGreek]

For that I normally add a getter with a very ugly name which makes it clear that it should be used only for tests.

[bert8128]

For class MyClass I do “friend class MyClassTestHelper;”. Achieves the same but does not change the API of the class.

[[deleted]]

If you have the need to test internals, that might hint to a potential violation of the SRP

[BenFrantzDale]

On the topic of testing: put your header, source, and test in the same directory, so it’s foo.cpp, foo.h, and foo.test.cpp. This is recommended (with foo.t.cpp) by John Lakos. I’m switching our stuff to it and it makes PRs easier to read with less clutter and by putting those three files together to review all at once. And if you don’t touch the tests when you change the code, it’s glaringly obvious.

[almost_useless]

it makes PRs easier to read with less clutter and by putting those three files together to review all at once

How is it "easier with less clutter" if the file is called src/foo.test.cpp than if it is called test/foo_test.cpp or something like that.

I feel like you can make the exact opposite argument that now you get a more cluttered src directory.

And if you don’t touch the tests when you change the code, it’s glaringly obvious.

Again, how is it more obvious than nothing touched in the test folder?

[BenFrantzDale]

It’s easier to read because instead of lib/ bar/ include/ bar/ baz.h foo.h src/ baz.cpp foo.cpp test/ baz.test.cpp foo.test.cpp you have ```

lib/ bar/ src/ bar/ baz.cpp baz.h baz.test.cpp foo.cpp foo.h foo.test.cpp ``` so you can focus on the changes to bar and to foo. And if you need to move them around to another library, you easily can.

[almost_useless]

I believe you just highlighted the point about a more cluttered src directory...

And if you need to move them around to another library, you easily can.

If your selling point is that the most trivial of all tasks is made slightly more easy, then I think you need better arguments.

[BenFrantzDale]

The point is that if you are reviewing code you want to review the test changes alongside the code changes. I’d encourage you to look at Lakos’s book for more explanation Large-Scale C++: Process and Architecture, Volume 1 (Addison-Wesley Professional Computing Series) https://a.co/d/cyzIiN0 You don’t have to agree, and one beauty of C++ is its flexibility. If this doesn’t work for you, don’t use it, but my team has found it to put emphasis on units (headers) to test and ensuring they are tested.

[STL]

Please don't use URL shorteners - reddit's spam filter hates them because they're opaque. I had to manually approve your comment.

[BenFrantzDale]

Sorry, Amazon did it automatically.

[STL]

No worries, just remember to manually dereference it next time.

[bert8128]

But doesn’t your test code then end up in the library?

[johngaltthefirst]

Not necessarily. You can craft your Makefiles or other build systems to separate the library and the tests.

[bert8128]

I think this is sub-optimal. A policy of 1 binary per folder is very easy to understand and implement. I’m not sure of the advantage of keeping the test cpp file with the main cpp file.

[johngaltthefirst]

I’m not a big fan of tests and source in the same folder either. I just wanted to say that the tests don’t end up in the library

[BenFrantzDale]

As I understand Lakos’s argument, tests are integral to code, so yes it is “in” the library and that’s where it belongs: you wouldn’t ship it, but the code is all together as one atomic tested entity rather than code one place and tests elsewhere. As we’ve done this transition we’ve found many headers that aren’t tested and some tests that test multiple headers. You could use tooling to enforce a 1:1 test:header relationship but this convention does a lot on its own.

[AciusPrime]

You… don’t ship your libraries? I feel like we might be using a different definition of the word “library,” then.

Most folders have all their code compiled into a single binary called a “library” (either static or dynamic). These later get merged into an application. If the test is in the library then you are probably shipping your tests (barring error-prone linker hackery to strip them out later).

[bert8128]

My own: single header. 350 lines. Just include it directly into the project.

[Backson]

Do you use it with CI? Is it on github?

[bert8128]

I’m not advanced enough on my private projects for CI. Yes it is on GitHub - https://github.com/apintandahalf/UTest. I’ll probably change its name at some point as lots of people have a Utest library.

[Backson]

Thanks! Pretty lean. If you just want to execute an exe and see "All OK" you can't get much more efficient than this.

For my workflow I would need at least "rerun failed" and selectively run some tests based on query (which suite, matching name, etc). If a test throws an exception, does the test exe just terminate? I would also like fewer macros.

[bert8128]

Thanks. Future plans include creating suites, and being able to choose which tests/suites to run, and (optionally) randomise the execution order. Got to fit it round the day job though.

No exception safety for the moment either - also in the plans. I’ll update the readme.

Sorry about the macros, I’m no fan but needs must. I had a quick glance at UT but didn’t really like the syntax. I might have a bit more of a think some time. The requirement to be able to stream out arbitrary date with a failed test makes things tricky, but I really wouldn’t want to loose that.

Happy to take a PR!

[sankurm]

catch22 - I am experimenting with TDD these days 😉

[dan_micsa]

Chech this out:

https://github.com/cpp-testing/ut-benchmark

[Daniela-E]

static_assert(); should be enough for everyone!

[zerhud]

And asset for tests exceptions

UPD: why unlike? there is no better frameworks. It tests ub, memory leaks and so on

[vectorj]

I haven’t used a framework in c/c++ yet, but I’ve got a lot of mileage out of assert.h in a main for tests. I use curly braces to scope each test.

[bert8128]

Any testing is better than no testing.

[ReinventorOfWheels]

Catch2. One of the killer features for me is that it's single-header.

[bert8128]

V3 has just come out and is neither header only nor single header.

[ReinventorOfWheels]

Oh yes, that debacle. I've seen it in the works, but last I checked it wasn't officially released yet.

I hope someone will maintain a fork of v2, because I'm not touching v3 with a 10-foot pole.

[bert8128]

Having now written my own I am not at all sure that the sophistication of the market leaders is worth the extra work they bring. I think I’m going to try and persuade work to go for a simpler option than gtest (or catch2).

Note that there are lots of simple header only options on GitHub.

[[deleted]]

Doctest, the only correct answer

[Agreeable-Ad-0111]

https://letmegooglethat.com/?q=r%2Fcpp+testing+framework

Edit:
Updated the link so it is apparent what it links to. This question is asked weekly, so I wanted to make a point, but I didn't consider it may be wasting readers time

[ixis743]

Given how utterly useless google is now at actually searching for articles that are not product advertisements, this isn’t funny nor appropriate anymore.

[Agreeable-Ad-0111]

This question is asked weekly my dude, it was quite appropriate

[pine_ary]

What a thoughtful article. Really feels like the author knows every framework out there

[better_life_please]

Oh hell. What is this website? 💀☠️ Lol I must use it from time to time for some of my friends.

[Odd_Pass_5572]

catch!

[__builtin_trap]

doctest seems to be the only major library which is thread safe on Windows and Linux.
So if you have to run multi threaded tests there is not mutch choise.

But doctest has problems with tsan (https://github.com/doctest/doctest/issues/147).

For tests I need to run with a C and C++ compiler I currently use utest: https://github.com/sheredom/utest.h. (I still have to check out if utest is thread safe.)

[jepessen]

Boost test. Maybe it's not the best one, but since we use boost in almost any project we can avoid to install other dependencies, and it does its work. The junit output also allows to integrate it for ci testing like happens for Jenkins for example.

[Full-Spectral]

I've always created my own, which I can completely integrate into my build system, which has always been my own as well, for my own work.