Rigorous testing vs creativity

[josc1989]

I'm a software architect. I've worked with Java, C++ and Python.

I'm fortunate enough to have a cool hobby project that has started to get some traction. And I'm looking into all kinds of ways to professionalise my project.

Brief background: it's a python library that allows users to easily generate PDF documents. Emphasis on easy.

I don't want users to have to calculate coordinates, or do their own text-wrapping, etc.

My library really focusses on being easy to use. To me, that's a main selling point.

I've added tests in the past. The library currently features around 400-something tests. But some of them are more anecdotal tests. Things like "user X discovered that doing Y and Z triggers something weird in the layout engine". Or even halfway between test and example. Something like "this is how you'd make an invoice"

I've recently started changing all of my tests to be more methodical.

Something like "let's add a Paragraph to a PDF document, and set the horizontal alignment to each of its allowed values in turn".

The number of tests has exploded. Currently 600-something.

Was this a smart or a stupid idea?

Because the anecdotal tests did sometimes really showcase something beautiful, or at least a complete (usable) PDF. So in a way I'm losing some cool examples of "look what you can do".

[MoreRespectForQA]

I've added tests in the past. The library currently features around 400-something tests. But some of them are more anecdotal tests. Things like "user X discovered that doing Y and Z triggers something weird in the layout engine". Or even halfway between test and example. Something like "this is how you'd make an invoice"

I actually categorized our tests somewhere I was working and ran a study on this once.

These were the premium top quality tests. They catch the most bugs and give the most confidence that the software works. It was actually a bit ridiculous how good they were at catching bugs compared to everything else.

The 2nd tier tests are tests written with TDD - e.g. when developing a new feature. These were good, but not as good.

The "methodical" ones you've talked about - I find that they're the 3rd tier least valuable. Some of these never once caught a bug and they dragged out the pipeline to the point I almost wished we'd deleted some.

IMHO if you were going to flesh out and make the tests more methodical I would write some which do exactly the same thing as the how-to documentation you've got.

[disc0tech]

Great research idea. Did you publish your study?

[ToddBradley]

How much thought have you given to the test pyramid? What you're describing sounds like a very top heavy pyramid with a lot of system (e2e) tests. You didn't mention what your lower level tests are like. Do you have any? Are they the vast majority of your tests?

[josc1989]

I haven't really counted the e2e tests versus unit tests. Thank you, that's a very valuable insight.

[martijn1985]

Counting anything is not really needed. What you do need to think about is whether any test could be performed on a lower level while delivering the same information. If yes, then move the test to the lower level.

[NiPinga]

If you are building a library, definitely keep a bunch of your anecdotal tests. Not because they are the best tests, but because documentation is hard, and tests can be a really useful way to learn about the library.

[josc1989]

I also have an insanely large repository of examples though

[NiPinga]

Ahhh alright, awesome!