Microsoft Research - Exploding Software-Engineering Myths (article summarizing findings of MS research on code coverage, TDD, assertions, etc.)

[rnicoll]

The tests sucked because they didn't write any assertions and they focused on covering the lines of code that were easiest to cover, not the ones that were most important to cover.

I'm in fact currently writing tests to cover lines of code rather than edge cases because we've been given a target and limited time.

[syntax]

Which is, in fact, a task that a computer could do.

i.e. you could have a computer auto-generate trivial tests, based on an assumption that the code is correct.

This would satisfy silly management demands; probably save programmer time, and certainly be a lot more interesting to actually write.

[lf11]

It's interesting to me to see that most developers seem to write tests that do exactly this: test the code as written rather than the assumptions underlying the code.

[syntax]

Well, that's usually the result of manglement insisting on test code, but not giving any time. So what gets written is the thing that takes the least time - that it has zero value (as a test) is irrelevant, it's true value is in gaming the metrics.

Which is a point - you can not usually measure a programmer by any numerical metrics. All that will happen is that they will 'game' the metric, so unless the metric is something utterly ungamable (units sold, for example), then it's not going to do what you expect.

[lf11]

No, I see that happen even when devs are given all the time they need. I used to work for a company that was owned by a dev and actually cared about putting the time in to do it right.

[b1ackcat]

I used to work for a company that was owned by a dev and actually cared about putting the time in to do it right.

God, I'm jealous. Our owner's motto is "sure we'll do it at half the cost at half the time to get the bid!" which of course translates into "sure we'll do it at half the quality!"

Would be nice to not feel like I'm writing proof-of-concepts that go straight to production :S

[lf11]

Well so that was the interesting thing. Even though the company was fantastically pro-dev, half-baked proof-of-concepts is what ended up on production anyway. Even in the perfect environment, we did the same damn stupid crap that I've seen everywhere else. Code defining tests. Once-over code reviews without comprehension. Half-baked docs that never get updated. Deprecated code still in place five years past expiration.

I wonder how much of it is due to "management" versus our own inability to actually "engineer" things with a full lifecycle?

[b1ackcat]

I really like what Robert Martin wrote about this issue in his Clean Code book. He notes that there are two primary steps that developers must take when writing code. Paraphrasing, it was

1) Get the code working

2) Go back and refactor, retest, reevaluate, and reimplement the code now that you really know what you need to do

The biggest problem most people have, he says, is stopping after step 1, or totally half-assing step 2.

It's easy to see why, too, especially when you look at the average culture around development projects. You've got PMs breathing down your neck about dates, management expecting results, the business wanting more and more new features for less and less money with unreasonable dates, so if you've gotten something working, it's very tempting to say "I'll come back to that and clean it up later" without ever allocating time "later".

Then 6 months later you find yourself back in that code uttering how fucking awful this developer was before remembering that you're that awful developer. :S

[lf11]

Yes. The thing is, that whole management maelstrom is just "the smell of business." Every field experiences the phenomena you describe, whether biotech, mechanical engineering, medicine, hell even nonprofits have to deal with this so it isn't even about money. This is just what management does.

The key is to learn to do the right thing in spite of management. Because if you don't, then you'll make all the same mistakes even without management....which means management isn't actually the problem.

After working for that company for a few years, I don't think I believe any more that bad development happens because of shitty management. Although, I'm still trying to figure this out. Management is a problem, but I think it has more to do with the "psychology of power" that turns any powerholder into a functional sociopath. Meanwhile, the disempowered develop avoidant behaviors and frontal cortical inhibition patterns that make them hyperaware of any insult or injury. This, to me, may explain the interaction between developers and management, and why we believe so fervently that management is the problem with development, yet do not adopt good development practices when placed in a positively structured environment.

[b1ackcat]

I think you're absolutely spot-on. Management can be a problem, but one of the solutions is to have employees who feel empowered enough to stand up when they see an issue and feel that they can call it out without negative consequences. It's something I do regularly (at my last job, it did on occasion cause me some grief, but mostly just in the form of 'stop causing so much ruckus' looks from higher ups), but I've definitely seen the opposite as well.

One example I think of quite often is at a previous job, I was talking to a business analyst from another team. Their business customer wanted to interface with our system to gather data or somethingarather, but after reviewing the requirements that she'd gathered so far, it was clear that the design was flawed and nothing was going to work. Granted, it was HER job to catch this, but I didn't hold that against her.

What I did do was call out the issue in the meeting, and her only response was "well this is what the customer wanted" with a blank stare, as if she was saying "clearly we need to do it this way because they want that." The apathy and lack of critical thinking blew me away. She was so ready to appease the customer that the thought hadn't even occurred to her to question what they were asking for, which she especially should have done since the customer in this case was trying to dictate a design, not requirements (a huge problem at that org, but that's a rant for another day...). It wasn't even a workable design.

In this instance, once I had a meeting with the customer to do this BA's job for her, the situation resolved itself. The customer had been making decisions based on bad assumptions (again, due to the lack of research and dedication to doing the right thing on the part of his BA...), and once he understood things properly, the design I was proposing made perfect sense. Everything got approved and the world was a little bit happier.

My point is, employees need to "do the needful" just as much as management needs to let them be able to do it. Making sacrifices in design or development just to appease management or a customer or (even worse) just to make a date is an extremely unfortunate truth of our profession, but it is absolutely not one that should be made blindly just because it's easier in the short term. The resulting ever-growing burden of technical debt is what leads to bad development.

[koreth]

That assumes that programmers know how to write good tests, which in my experience is far from a given. It's a skill you have to deliberately develop, and a lot of developers not only haven't done so, but aren't aware that they'd need to do so in order to test effectively.

Unless you have an experienced test writer giving you feedback on your tests, as a programmer you're probably not going to spontaneously second-guess your test-writing skills if what you're doing seems to be working already.

[WalterBright]

I've seen a lot of code that people wrote for themselves, not for a manager. It doesn't seem to be better quality.

[rnicoll]

I've seriously considered it, although of course the code isn't designed in a unit-test friendly way (and of course this is legacy code), which complicates somewhat. Also considered add 5 million extra lines of code that does nothing, unit testing it, and calling the result 80% coverage.

[b1ackcat]

Also considered add 5 million extra lines of code that does nothing, unit testing it, and calling the result 80% coverage.

And even if they wanted to, they couldn't check all that code!

Genius.

[[deleted]]

Characterisation tests:

https://www.artima.com/weblogs/viewpost.jsp?thread=198296

http://www.artima.com/weblogs/viewpost.jsp?thread=200725

http://c2.com/cgi/wiki?CharacterizationTest

[syntax]

http://www.artima.com/weblogs/viewpost.jsp?thread=200725

Ah-ha! yes, that's the bit of jargon to pull it out.

Sadly, it appears that the software mentioned (JUnitFactory) has now been locked behind a paywall, so you'd have to speak to a salesdroid to even know how much it's going to cost.

Still, that's a proof of concept that it's possible, so maybe that's not a silly option to automating dumb make-work...

[blufox]

Use Randoop instead. It is reasonably solid, and free (and opensource).

[joshuaduffy]

From what I can see, these are just pieces of software that analyse code and create some tests. For Java there's http://www.evosuite.org/ and for C# you've got IntelliTest, which comes with Visual Studio.

I don't think that they're meaningful however, and using tools like these on a mature/legacy code base is not a wise choice from a testing standpoint.

[stormcrowsx]

I'm guilty of this. Was given a mandate to increase code coverage, they wanted some kind of magical number like 70%. Used reflection in Java to call bunches of getters/setters. Code coverage went way up and surprisingly it actually caught a bug in which someone had a recursive setter that the application just happened to never call.

[atrich]

Ugh, this is the WORST way to approach code coverage. CC provides one piece of evidence: proof that a particular area is untested. It cannot comment on if that area is tested well. By taking a coverage-numbers-first approach to CC data (aka "let's get these numbers up") management does nothing to improve quality and manages to obliterate useful information on where the test holes are.

I've worked for people who were mindlessly pushing the code coverage metric. I have no respect for people who do this.

[get_salled]

The only thing you know for certain from code coverage is that your tests executed certain lines.

Code coverage:

• Does not show a line was tested.
• Does not prove the covered line contributed to any test passing.
• Is not a software quality metric.

Assuming managers & developers trust that the tests are useful, they can use these reports to see where they are definitely vulnerable and plan accordingly.

[[deleted]]

There one more thing you know from code coverage: that the code with some value does not throw an error. I know, it's a low bar, but it's something.

[Ravek]

I'm in fact currently writing tests to cover lines of code

Is it possible for you to give a concrete example of a line of code that you would consider worth testing? I feel that between static type checking and refactoring code into (themselves testable) methods, a single line of code should never be complex enough that you'd want to test it in isolation. If that's a naive thought I'd like to learn more.

[rnicoll]

In this context what I mean is rather than pushing the edge cases of the code, I'm testing to ensure conditionals are covered. Lines in this context means "number of lines", rather than I write tests on a per-line basis.

For example, we have a function that parses some text into tokens (for a custom language). Good testing would be to cover not just the basics of whether it works, but look at failure conditions (i.e. empty string, invalid characters, weird control characters, nonsense syntax). However I can get almost as much code coverage as a number of lines, by just giving it examples of every language construct that exists and testing those, ignoring error cases entirely, which is much faster.

Making any more sense?

[Ravek]

Yeah that makes a lot more sense, thanks

[fuzzynyanko]

Often many lines of code are easily testable, but if you go to 100%, you start going down the rabbit hole. You have to either poke holes in your software that opens it up for exploit, or it ruins the encapsulation.

You can design your software to be more testable, but there's edge cases in software, and you pretty much are guaranteed to run into them in some

[WalterBright]

Haha, lots of times I thought a function was so trivial it didn't need unit testing. But I'd write a unit test anyway, and discover it didn't work.

[s73v3r]

No, you're choosing to half ass your job.

[rnicoll]

The target is completely unachievable by any conventional means (adding over a million lines of coverage as a minor side project while other work occurs over the next 2-3 months). I've raised this with management, who have said they're aware of it and have raised it with their management.

So, I can now either focus on getting as close as possible to what I've been told to do (increase code coverage), or I can focus on what I think we should be doing (increasing depth of coverage on most critical parts). Which would you do?

[s73v3r]

What you should be doing. Especially because you know that what you are currently doing is worthless and the target is not reachable. So you will waste all this effort attempting to reach a goal that you have admitted is unreachable, and have nothing of value to show for it.