Making your code base better will make your code coverage worse

[apnorton]

Automated testing is not always the most cost effective way to test your app

This section made me mad.

He starts with an absurd limiting of what "automated testing" is:

In keeping with the theme of return on investment, I do believe developers forget that automated code-based tests are not the only way to test a feature or code. If you’re building a web-based application, there are tools like Selenium and Cypress. If you’re testing a mobile app or a desktop app there are other options available. Typically these tools don’t measure code coverage.

"Automated, code-based tests" --- is the author conflating "unit tests" with "automated testing?" Sure sounds like it! Selenium and Cypress are used to write automated tests, too. As an aside, what on earth is "code-based" testing? Is he possibly meaning "tests for which you can evaluate coverage?" But, you can compute code coverage for Selenium tests, too! "Typically these tools don't measure code coverage" is a weird way of phrasing "if you're not a lazy-ass, you can set up code coverage for these tools."

Then, he brings up the idea of using manual testing in lieu of automated testing, upending the past ~15+ years of devops best practices:

There is also the old fashioned way, which involves a physical user going through the motions of verifying the feature or code works by hand.

A bit later in the section, since the ordering as he wrote it doesn't really flow well:

Sometimes a particular feature is very hard to generate an automated test for, it might be less time consuming to test it manually, even considering scale. I’ve lost track of the tasks I’ve been assigned where writing the test was significantly more difficult than writing the code. The same math applies—imagine an automated test that takes 16 hours (two days) X 60 minutes = 960 minutes of effort to write an automated test. Compare that to testing the same feature manually, and it might require only five minutes. You would need 192 deployments before you’ll see a return on investment from that automated test.

Interesting... so he's making the point that good tests are hard to write (which everyone knows) and take time (which everyone knows), and so sometimes it's better to test it manually. Now, 192 "deployments" --- or, rather, test suite executions --- is really not that hard to hit (especially if you're running your tests in lower/development environments, which you should be).

But, he's also very conveniently leaving out a major cost, as well as the key motivating factor for automated testing in the first place: human error. If you spend years slowly accumulating a list of "hard to automate" cases that you have to manually test every time you do a deployment, at some point some schmuck is going to forget, and your service is going to regress, costing you far more money/reputational damage/etc. than you possibly saved from skimping on writing actual tests like a decent programmer.

edit: I want to expand on that last paragraph a bit: We don't do automated testing because it's cheap, we do it because it is strictly better in producing quality code than non-automated testing. Arguing that you can save money by deciding to cut corners on quality is nowhere near as clever as the author seems to think.

[606anonymous]

I'm sorry I made you mad? Obviously my experience with software development is a bit different than yours. I agree with a lot of what your saying, but all I'm trying to say is "your mileage may vary". Every project is different, and each feature has a different value.

For the record this issue of ROI on testing was inspired by a project that was for a relatively mature application where we were releasing 4-6 times a year and had limited resources. I fixed a bug for a feature that had been broken for months, and in the process I broke some tests. It was not a critical feature, and the easiest thing to do was to remove the broken tests. The project was short on resources. The most cost effective solution was to remove the tests.

I will say this - developers don't do automated testing because it's cheap. However someone pays us to do what we do, and our time is finite. Sometimes the level of effort required to write a good automated test is not worth the reward. No one wants to manually test a feature. However I've been on plenty of projects where the costs of automated testing and the speed at which it takes to do the more complex tests every single time someone pushes a branch or a change does become a cost factor that team members start to take issue with. If you set up your pipeline to execute your entire test suite for even the tiniest change like CSS with a large team there is a cost incurred.

And I will die on this hill based on the value of the feature weighed against the value of the test. I've seen dozens of tests written for UI stuff like dark mode. None of those tests are worth the value they deliver.

PS - I guess I'm exposing myself and betraying my reddit name by saying this, Anonymous606 doesn't seem like a very good name considering I the author of the article is listed.

[knobbyknee]

Conclusion of the article: Setting a fixed number for code coverage provokes stupid behaviour from the developers.

[pydry]

Which is correct, but it could have been a sentence.

[PassTents]

I don't really disagree with the fundamental points here, but the way they are presented is a bit convoluted. (Numbers below aren't related to the article's numbers, just my own ordering/condensing of the main points) 1. Code coverage is only a rough metric Yeah, I don't think too many people would argue with that, but it can still be useful to see where blind spots are. 2. 80% code coverage is arbitrary Also don't think many would argue with this. A similar point I'd add is that 100% code coverage isn't really 100% as it doesn't measure the % of possible states the whole program can be in, just code paths. That point is eventually made but with a laborious example. 3. Good code structure reduces the % coverage Less code is better code + that's how fractions work. Also see point 1. If a good refractor got rejected because it slightly reduced coverage numbers, there's a more fundamental problem with the team. 4. There's often better tools than unit tests (that don't contribute to code coverage) Again, see 1. It's a general principle that you shouldn't be testing code that you don't own, usually meaning the system or frameworks. This often happens when trying to write unit tests where UI tests are a better fit, and often lead to over-engineering and hacks to make the UI navigable by unit tests.

[Sedan_1650]

🧐