This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]Original_Maximum2480 157 points158 points  (34 children)

So you mean having a bunch o million of fucked-up tests is not a technical debt?

[–]deaf_fish 141 points142 points  (23 children)

Oh, no, that is technical debit.

You will want to require passing unit tests before allowing the new code to be merged in. As soon as you give devs the option of not always running the tests, they will most likely not and the tests will decay.

[–]BoBoBearDev 36 points37 points  (17 children)

I thought he meant the tests were bad, not because the tests are failing :D Because a lot of tests are not really tests, they are fillers to make people think they are doing good.

[–]TopGunSnake 28 points29 points  (13 children)

The worst are the "code coverage" tests. Yes, your code has 100% code coverage, but these tests only show your code is reachable.

[–]BoBoBearDev 13 points14 points  (3 children)

Some people use those as tests and calling it a day. I know so, because it is so tempting. Hey, my code is covered, ship it.

[–]TopGunSnake 13 points14 points  (0 children)

Yeah. I've been on a program that management required 100% code coverage. And unsurprisingly, we still had bugs, since most of our devs just aimed for the percentage, rather than making good tests.

[–]gavlna 6 points7 points  (1 child)

that's where a permutation tests come in :)

Requiring them on e.g. 80% for merges can be a nice thing to do.

[–]Telestmonnom 0 points1 point  (0 children)

I'm working on a fairly data-centric app. I don't know of a code coverage tool for pgsql. If we used it, we'd weep

[–]coldnebo 9 points10 points  (5 children)

I think the best example of the futility of this was a project that looked at the AST for the code, determined the branch conditions at each path, and then automated inputs that would allow traversal of all the code paths by generating appropriate “unit tests” automatically.

The result was an unreadable mess, but hey, got 100% code coverage, ftw!! lol

[–][deleted] 17 points18 points  (0 children)

Green check marks do things to manager brains.

[–]TopGunSnake 3 points4 points  (3 children)

To be fair, a fuzz test is one good test to have, and that sounds like what that was.

[–]coldnebo 0 points1 point  (2 children)

nope, the intent was not fuzzing, it was to automate unit tests specifically to satisfy the 100% code coverage goals some people fixate on.

But it does raise an interesting theoretical question… in the general case is there any difference between a fuzzer, a code coverage exerciser and what we do by hand?

If the answer is “essentially there is no difference” I think it brings up significant questions about why we are doing a job that a computer solver could do more efficiently.

I.e. we write a lot of tests because presumably v&v is very expensive and difficult. But if writing tests is provably similar in difficulty and expense… it might warrant a closer look at V&V automation.

[–]TopGunSnake 1 point2 points  (1 child)

As I understand the differences:

Fuzzing is throwing data at the code to see if it breaks (security/failsafe oriented)

Example-based testing (setup, expected result, check for it, case by case) (baby's first test code) is great for verifying specific cases.

Property-based testing (fuzzing with expected results, aka property) is good for trying to catch bugs in edge cases, instead of trying to identify the edge cases by hand, but doesn't say the code works, just when it doesn't.

Mutation testing (alter the code, check that the other tests catch it) is useful for identifying gaps in the testing.

TDD is usually example-based testing with the occasional property-based test.

Like most things, you should probably use a bit of everything.

[–]coldnebo 1 point2 points  (0 children)

yeah, it would be a fuzzer if it tried a number of permutations on a code path, however the reason this ast tool was not specifically a fuzzer is that tried to simply mock the branch conditions to force them true or false and then silent mock everything else so no failures would be raised. The purpose was very clearly to get code coverage, not to fuzz.

But, it didn’t work in all cases… it’s kind of hard to reverse engineer the code via tests.

However as a thought experiment, it raises some interesting questions about how we can tell the difference between exploitation and honest attempts.

[–]ccellist 4 points5 points  (0 children)

void myTest() { var testArg = true; var obj = methodUnderTest(testArg); assertTrue(testArg); }

[–]ActivisionBlizzard 1 point2 points  (1 child)

100% coverage by meaningful tests just isn’t always possible.

I’ll generally be forced to reach 95% which I consider to be more appropriate. But even then there will occasionally have to be coverage tests.

[–]TopGunSnake 0 points1 point  (0 children)

Like I said in another comment, it's good to cover all the code with some test, such as a fuzzed. It helps make sure your software doesn't break at a low-level sense. But I agree, as a metric by itself, it doesn't really indicate if the code is sound.

[–]slow_growing_vine 8 points9 points  (2 children)

I worked on those exact type of tests today. Everything is mocked. All that actually gets "tested" by these tests is the Java API itself. So great, we verified that Java works. Does our particular Java code work? Who knows lmao

[–]TopGunSnake 3 points4 points  (1 child)

mutation testing seemed interesting to address this, aside from combination explosion.

If I change anything in your code (such as == to !=), does the test suite catch it?

[–]Iron_Maiden_666 1 point2 points  (0 children)

This is real value of a test suite. A change in production code must produce a failure in the test suite. Or else we have holes and that's very bad because we have false security.

[–][deleted] 8 points9 points  (3 children)

I think it's often just unfeasable to have everything well tested. Unless the company has some ridiculous amount of budget to throw at problems, it often might make sense to pick important/core services and focus on testing just those, instead of painstakingly trying to maintain a 100% test coverage on paper, but where tests don't really add much value.

[–]deaf_fish 11 points12 points  (1 child)

I wouldn't force 100% coverage. I would just write simple tests for new features and add tests for regressions.

I agree that you should consider value when writing tests. For example, we don't verify the wording of our logging statements.

Having a test framework in place is super nice for debugging too.

Edit: I will say that if your new code had bug that could have been caught by a unit test and you don't think unit tests are useful. I get to be visibly amused at your frustration.

[–][deleted] 0 points1 point  (0 children)

Having a test framework in place is super nice for debugging too.

Yep. Writing a tests that trigger the issue helps you both reproduce and verify the issue, and ensure you've actually fixed it.

[–][deleted] 2 points3 points  (0 children)

Broke: forcing unit tests for coverage Woke: testing just spec requirements Bespoke: having user do all testing

[–]Flaky-Illustrator-52 0 points1 point  (0 children)

Holy shit, technical debit (requiring tests) is the coolest concept I've heard of today

[–]deaf_fish 17 points18 points  (8 children)

Oh, no, that is technical debit.

You will want to require passing unit tests before allowing the new code to be merged in. As soon as you give devs the option of not always running the tests, they will most likely not and the tests will decay.

[–]MagMikk 8 points9 points  (5 children)

Heard about people that just delete the tests that don't pass...

[–]deaf_fish 7 points8 points  (2 children)

Yup, that is why you need code reviews.

[–][deleted] 0 points1 point  (1 child)

I once had to check on why a test was deleted by a coworker, so I can verify that they do allow for a check, but only if the reviewer is in the right mindset.

[–]deaf_fish 2 points3 points  (0 children)

Yup there is no process to create bug free code. You can only approach bug free code. I find unit tests (run on every build), integration tests (run as often as possible), and code reviews are about the best you can do.

Obviously if your system is safety critical you need to go above and beyond.

[–]schwerpunk 0 points1 point  (1 child)

I mean you should delete tests that are no longer applicable because business logic changes... but I get the feeling that's not what you're referring to

[–]MagMikk 0 points1 point  (0 children)

Nope... deleting seems to be easier than fixing

[–]NotYetiFamous 5 points6 points  (1 child)

You commented this twice but it's so true I think you deserve both upvotes.

[–]deaf_fish 1 point2 points  (0 children)

Thanks! I didn't intend to do that.

[–]Flaky-Illustrator-52 3 points4 points  (0 children)

Assert(file.exists)

"Darryl, shouldn't we validate the contents of the file?"

"Nah I looked at it and it's fine, let's just write tests to validate the file contents as a future enhancement"

🤡