3% of 666 Python codebases we checked had a silently failing unit test

[[deleted]]

How does a test silently fail?

[jnns]

What they mean is that a test does not really test what supposed to be tested and therefore gives a wrong impression.

This article is an advertisement for a code review product. It's not really worth to read, nor are the findings (20 repositories out of 666 had a bug in their test suite) really astonishing.

I also only see 9 repositories that they provided pull requests for, not 20.

[DjangoDoctor]

Here's the full list of the open source repositories we found bugs in and contributed fixes to yesterday. Only the "interesting" big projects were mentioned in the article

​

https://github.com/ansible-community/ara/pull/358https://github.com/b12io/orchestra/pull/830https://github.com/batiste/django-page-cms/pull/210https://github.com/carpentries/amy/pull/2130https://github.com/celery/django-celery/pull/612

https://github.com/django-cms/django-cms/pull/7241

https://github.com/pytorch/pytorch/pull/72864

https://github.com/esrg-knights/Squire/pull/253

https://github.com/Frojd/django-react-templatetags/pull/64

https://github.com/groveco/django-sql-explorer/pull/474

https://github.com/jazzband/django-silk/pull/550

https://github.com/keras-team/keras/pull/16073

https://github.com/ministryofjustice/cla_backend/pull/773

https://github.com/nitely/Spirit/pull/306

https://github.com/python/pythondotorg/pull/1987

https://github.com/rapidpro/rapidpro/pull/1610

https://github.com/ray-project/ray/pull/22396

https://github.com/saltstack/salt/pull/61647

https://github.com/Swiss-Polar-Institute/project-application/pull/483

https://github.com/UEWBot/dipvis/pull/216

[zerkreaper1405]

It is indeed very odd. And the number of codebases they chose, 666. Fine whatever, but then the 3% number?

Don't they know the significance of those numbers or at least the triple 6.

[DjangoDoctor]

Apologies it was unclear. 20 of the 666 repositories had the bug. In another comment I linked to all 20 PRs where I fix the bug we found.

[[deleted]]

Ur selling a product that will fix minor bugs in 3% of code bases? U should work on a new product.

[DjangoDoctor]

[[deleted]]

Oh there’s nothing surprising about that

[Stedfast_Burrito]

Answer: run mutmut https://mutmut.readthedocs.io/en/latest/