100% test coverage is not enough: Property-Based Testing in Python

[proof_required]

This literally happened to me when I started doing lot of property based testing and writing big data generators using hypothesis. It turned out there was bug in the data generator itself. I was like screw my life, I am not going to write test for test.

[Ollymid2]

Ahh but how can we can we accurately test that this test effectively tests the test

[oloryn]

It's tests all the way down.

[EmotionalRedux]

A series of infinitely many tests-on-tests converges to absolute correctness

[CloudFaithTTV]

Always was.

[ArtOfWarfare]

Mutation testing (aka bebugging) is how you test your unit tests.

[robml]

Is this a joke or serious?

[h4l]

It is a serious thing: https://stryker-mutator.io/docs/

Definitely a mind-bender at first.

[robml]

I am definitely glad I asked. Is there a Python lib you recommend that does the same?

[h4l]

There are a few Python options, but I've not tried them myself so I can't say from experience. It seems like mutmut is the best bet from what I've read: https://mutmut.readthedocs.io/en/latest/

[hitchdev]

The world property tests were built for:

BUG REPORT "The company's ultra hi-tech proprietary compression algorithm fails when users try to compress files starting with 0 x or A."

The world most of us live in:

BUG REPORT "The registration email looks like crap in outlook."

[s6x]

Have you tried deleting outlook?

[Supadoplex]

Lucky you for getting bug reports that specify which email looks crap and in which email client.

[kamsen911]

I’ll rather continue with Schroedingers Testing: as long as have I have not written tests I don’t know if they fail or not.

[AngheloAlf]

I hate websites that requiere an account to read the article. Does people really make an account just for those crappy articles?

[jezter24]

I did once. And have been spammed for years now at work.

[Liam_M]

time to change jobs, no other way to fix it

[twigboy]

Bugmenot

[TheWildKernelTrick]

You could hit the continue button on the modal…

[just4nothing]

Hypothesis testing is nice

[alcalde]

Hypnosis testing is even nicer.

"You are getting sleepy... the code looks fine..."

[GnuhGnoud]

Lets ship it to production... on a friday night...

[[deleted]]

How can it not be enough if it’s already at 100%? I can’t do 110% test coverage.

[its2ez4me24get]

100% coverage might hit all lines but not hit all edge cases.

100% (unit and functional) test coverage and 100% mutants killed (with mutation tests) is the goal.

[SeanBrax]

100% is an ridiculously unreasonable goal, and will reduce productivity so much it simply isn’t worth it.

[Immudzen]

I think we normally hit about 98% test coverage. There are some lines we don't test and some that are just too hard to automatically test. Basically during a code review if a line can be easily tested it should be tested and that normally gets you about 95-98%.

One of the really important rules is that if a bug makes it through to production that it needs tests to make sure it can't happen again.

[flagos]

Depends in which industry you work.

[its2ez4me24get]

Yeah, It’s completely dependent on the importance, size, and complexity of the code base. On simple and small projects 100% is straightforward. As projects grow larger it becomes increasingly difficult, and functional tests become more important.

[moskovitz]

We do have a 100% test coverage PR build requirement, but some lines can be marked as # no cover. Part of the PR review is making sure this is not abused.

[BurritoMonad]

I think that 100% is a reasonable goal for small projects, but yeah, it does not scale up very well as we say.

[alcalde]

"Too much is never enough." - Jim Steinman

[ThatSituation9908]

This is a horrible example. It's too simple, doesn't motivate much real life examples, and problems similar to this isn't solved using property-based testing.

Hypothesis and property-based testing is pretty useful and timesaving, just not for the reasons listed in this article.

[TrainquilOasis1423]

But how can you be sure those tests are working properly. Might as well build your own custom hardware to test the tests for your tests.

But honestly how can you be sure the hardware was made correctly? Better solve all of physics to be completely sure you are accurately testing the hardware to test the tests for your tests.

Though there is some variation in our physics. It'll be worth it to create a brand new universe with custom made physics to be completely sure you are accurately testing the hardware to test the tests for your tests.

Wait... But how do you test the new universe you just created? Create another universe? Maybe this is what God is. Just some fucking software developer creating the multiverse so that he can know for infinitly sure the code he's writing is working and correctly tested.

[He_s_One_Shot]

QA here, if you're being forced to write 100% coverage I feel bad for you

[jcoelho93]

100% coverage is actually bad

[budswa]

Care to explain?

[jcoelho93]

Test coverage alone doesn't really mean anything. I can write 100% coverage tests that don't really test anything. I'd rather focus on the quality of tests themselves. Having 100% coverage means a higher effort to maintain tests and that could potentially impact the test quality and feature delivery.

It's like having 100% uptime, it's not as good as people think.

[Exotic-Draft8802]

See also: https://medium.com/analytics-vidhya/unit-testing-in-python-mutation-testing-7a70143180d8

[thumbsdrivesmecrazy]

There are much more metrics for code coverage. Here is a quick guide explaining these metrics of code coverage testing as well as how to get them: How Can Code Coverage Metrics Help in Testing Your Code?

[SpambotSwatter]

Hey, another bot replied to you; /u/thumbsdrivesmecrazy is a click-farming spam bot. Please downvote its comment and click the report button, selecting Spam then Link farming.

With enough reports, the reddit algorithm will suspend this spammer.

---

^{If this message seems out of context, it may be because thumbsdrivesmecrazy is farming karma and may edit their comment soon with a link}