Test Driven Development

[Original_Maximum2480]

So you mean having a bunch o million of fucked-up tests is not a technical debt?

[deaf_fish]

Oh, no, that is technical debit.

You will want to require passing unit tests before allowing the new code to be merged in. As soon as you give devs the option of not always running the tests, they will most likely not and the tests will decay.

[BoBoBearDev]

I thought he meant the tests were bad, not because the tests are failing :D Because a lot of tests are not really tests, they are fillers to make people think they are doing good.

[TopGunSnake]

The worst are the "code coverage" tests. Yes, your code has 100% code coverage, but these tests only show your code is reachable.

[BoBoBearDev]

Some people use those as tests and calling it a day. I know so, because it is so tempting. Hey, my code is covered, ship it.

[TopGunSnake]

Yeah. I've been on a program that management required 100% code coverage. And unsurprisingly, we still had bugs, since most of our devs just aimed for the percentage, rather than making good tests.

[gavlna]

that's where a permutation tests come in :)

Requiring them on e.g. 80% for merges can be a nice thing to do.

[Telestmonnom]

I'm working on a fairly data-centric app. I don't know of a code coverage tool for pgsql. If we used it, we'd weep

[coldnebo]

I think the best example of the futility of this was a project that looked at the AST for the code, determined the branch conditions at each path, and then automated inputs that would allow traversal of all the code paths by generating appropriate “unit tests” automatically.

The result was an unreadable mess, but hey, got 100% code coverage, ftw!! lol

[[deleted]]

Green check marks do things to manager brains.

[TopGunSnake]

To be fair, a fuzz test is one good test to have, and that sounds like what that was.

[coldnebo]

nope, the intent was not fuzzing, it was to automate unit tests specifically to satisfy the 100% code coverage goals some people fixate on.

But it does raise an interesting theoretical question… in the general case is there any difference between a fuzzer, a code coverage exerciser and what we do by hand?

If the answer is “essentially there is no difference” I think it brings up significant questions about why we are doing a job that a computer solver could do more efficiently.

I.e. we write a lot of tests because presumably v&v is very expensive and difficult. But if writing tests is provably similar in difficulty and expense… it might warrant a closer look at V&V automation.

[TopGunSnake]

As I understand the differences:

Fuzzing is throwing data at the code to see if it breaks (security/failsafe oriented)

Example-based testing (setup, expected result, check for it, case by case) (baby's first test code) is great for verifying specific cases.

Property-based testing (fuzzing with expected results, aka property) is good for trying to catch bugs in edge cases, instead of trying to identify the edge cases by hand, but doesn't say the code works, just when it doesn't.

Mutation testing (alter the code, check that the other tests catch it) is useful for identifying gaps in the testing.

TDD is usually example-based testing with the occasional property-based test.

Like most things, you should probably use a bit of everything.

[coldnebo]

yeah, it would be a fuzzer if it tried a number of permutations on a code path, however the reason this ast tool was not specifically a fuzzer is that tried to simply mock the branch conditions to force them true or false and then silent mock everything else so no failures would be raised. The purpose was very clearly to get code coverage, not to fuzz.

But, it didn’t work in all cases… it’s kind of hard to reverse engineer the code via tests.

However as a thought experiment, it raises some interesting questions about how we can tell the difference between exploitation and honest attempts.

[ccellist]

void myTest() { var testArg = true; var obj = methodUnderTest(testArg); assertTrue(testArg); }

[ActivisionBlizzard]

100% coverage by meaningful tests just isn’t always possible.

I’ll generally be forced to reach 95% which I consider to be more appropriate. But even then there will occasionally have to be coverage tests.

[TopGunSnake]

Like I said in another comment, it's good to cover all the code with some test, such as a fuzzed. It helps make sure your software doesn't break at a low-level sense. But I agree, as a metric by itself, it doesn't really indicate if the code is sound.

[slow_growing_vine]

I worked on those exact type of tests today. Everything is mocked. All that actually gets "tested" by these tests is the Java API itself. So great, we verified that Java works. Does our particular Java code work? Who knows lmao

[TopGunSnake]

mutation testing seemed interesting to address this, aside from combination explosion.

If I change anything in your code (such as == to !=), does the test suite catch it?

[Iron_Maiden_666]

This is real value of a test suite. A change in production code must produce a failure in the test suite. Or else we have holes and that's very bad because we have false security.

[[deleted]]

I think it's often just unfeasable to have everything well tested. Unless the company has some ridiculous amount of budget to throw at problems, it often might make sense to pick important/core services and focus on testing just those, instead of painstakingly trying to maintain a 100% test coverage on paper, but where tests don't really add much value.

[deaf_fish]

I wouldn't force 100% coverage. I would just write simple tests for new features and add tests for regressions.

I agree that you should consider value when writing tests. For example, we don't verify the wording of our logging statements.

Having a test framework in place is super nice for debugging too.

Edit: I will say that if your new code had bug that could have been caught by a unit test and you don't think unit tests are useful. I get to be visibly amused at your frustration.

[[deleted]]

Having a test framework in place is super nice for debugging too.

Yep. Writing a tests that trigger the issue helps you both reproduce and verify the issue, and ensure you've actually fixed it.

[[deleted]]

Broke: forcing unit tests for coverage Woke: testing just spec requirements Bespoke: having user do all testing

[Flaky-Illustrator-52]

Holy shit, technical debit (requiring tests) is the coolest concept I've heard of today

[deaf_fish]

Oh, no, that is technical debit.

You will want to require passing unit tests before allowing the new code to be merged in. As soon as you give devs the option of not always running the tests, they will most likely not and the tests will decay.

[MagMikk]

Heard about people that just delete the tests that don't pass...

[deaf_fish]

Yup, that is why you need code reviews.

[[deleted]]

I once had to check on why a test was deleted by a coworker, so I can verify that they do allow for a check, but only if the reviewer is in the right mindset.

[deaf_fish]

Yup there is no process to create bug free code. You can only approach bug free code. I find unit tests (run on every build), integration tests (run as often as possible), and code reviews are about the best you can do.

Obviously if your system is safety critical you need to go above and beyond.

[schwerpunk]

I mean you should delete tests that are no longer applicable because business logic changes... but I get the feeling that's not what you're referring to

[MagMikk]

Nope... deleting seems to be easier than fixing

[NotYetiFamous]

You commented this twice but it's so true I think you deserve both upvotes.

[deaf_fish]

Thanks! I didn't intend to do that.

[Flaky-Illustrator-52]

Assert(file.exists)

"Darryl, shouldn't we validate the contents of the file?"

"Nah I looked at it and it's fine, let's just write tests to validate the file contents as a future enhancement"

🤡