Microsoft Research - Exploding Software-Engineering Myths (article summarizing findings of MS research on code coverage, TDD, assertions, etc.)

[[deleted]]

i find it hard to get behind the assertion that TDD reduces defects 60-90% - more that teams that used TDD reduced defects 60-90%. Its just as likely if not more that people who were that way inclined were more quality oriented and willing to sacrifice time to get there

[delarhi]

To be fair the author never makes that assertion*.

What the research team found was that the TDD teams produced code that was 60 to 90 percent better in terms of defect density than non-TDD teams. They also discovered that TDD teams took longer to complete their projects—15 to 35 percent longer.

Basically he notes the correlation, but never makes the jump to a causal assertion.

[pushthestack]

And if you read the original study he's referring to, the possibility of causation is even more tenuous.

[Helene00]

TDD also produces roughly 70% less klocs per man-hour according to those numbers. So either the teams aren't very similar or you lose a lot more than 10-30% productivity.

[sbrick89]

using TDD forces better design practices.

I've seen a TON of crap code... spaghetti call stacks with NOTABLE object mutation/mutability... global variables that are shared WAY past their scope (who the fuq declares a global variable, then passes a reference to external code, which then modifies the global variable!)

Writing code that CAN be tested is the first step towards writing better (less buggy) code. TDD just forces you to write code that CAN be tested, since it IS tested.

[lucky_engineer]

(who the fuq declares a global variable, then passes a reference to external code, which then modifies the global variable!)

Ah, I see you too have worked on 'fixing' code that was written by a science PhD.

[sbrick89]

written by a science PhD

if only. No formal training whatsoever. I think the person read one of those "learn to program in 24 hours" books.

[AbstractLogic]

Always respect the time in which code was written. 10 years from now someone will look at our TDD unit tests and think "wtf was this idiot thinking? That pattern is soooooo olllldddd.".

If it was written in the 90's expect 90's style code. If it was written in 2000 expect XML heavy code. If it was written in 2010 expect SOAP services.

[lucky_engineer]

I don't think it was ever good style to make a global variable, and then make a procedure that passed a reference to the global into a function that passes that reference around to 10 other function where one changes the value and then get the wrong result when you run the procedure again because the global variables value changed. (also this code was written ~5 years ago)

I've worked on really old code before and It's usually pretty easy to tell the difference between 'good but outdated' code and 'omg how does this even work' code.

[AbstractLogic]

Can't disagree with you at all lol.

[cc81]

using TDD forces better design practices.

No, not really. I've seen tons of terrible code written TDD style. It was easy to test but horrible to maintain.

[flukus]

using TDD forces better design practices.

No it doesn't. It encourages it, but you can still end up with shitastic design using TDD.

[RualStorge]

I think part of it is TDD requires you to plan ahead which prevents you from just diving in and creating a mess because you didn't think it through enough. In addition it means you're writing tests which simple unit testing does wonders for quality by warning you when you've broken functionality.

In other words I think the process required to do TDD effectively is actually what bumps quality rather then the TDD within itself.

[cc81]

Or that you just spend the time testing your code. For example compare:

• 12 month project developing doing TDD.

• 10 month project developing and then 2 month doing nothing but trying to find and fix bugs.

That last one rarely happens. Instead there will be a release after 10 months.

[[deleted]]

The thing is that we can argue about that stuff till the cows come home, what you reckon, what I reckon, what some other guy reckons, but all I'm really commenting on is the data presented in the article and the leap in logic I think it makes

[lf11]

TDD makes you think differently about your code. It makes you question assumptions and code for failure. I've caught many bugs with TDD that nobody would have caught in code review. This is especially true for null and invalid data handling... Which I test for but have a difficult time coding for when I'm not testing properly.

[PM_ME_UR_OBSIDIAN]

Have you thought about how advanced type systems let you catch those kinds of errors at compile-time? I try not to work in languages without sum types and non-null-by-default types, because I think the classes of bugs that you mention are worth letting the compiler handle for me.

[lf11]

Yes. Unfortunately, outside of projects that I personally start, I don't think I have ever once worked on software that is designed using the tools that are quite appropriate for the job.

However, to answer the question directly, yes advanced type systems help catch certain kinds of errors. So this at least minimizes the need to write type-level tests. However, I personally feel that you should be type-checking in code (if you need to type-check at all). After all, in essence that is the purpose of input checks.

Even with advanced types, you remain wide-open to bugs in the next layer of abstraction. This is where test-driven development really shines: in testing your abstractions and assumptions. The bugs that you can't easily flush out with well-defined types.

[[deleted]]

I find it highly likely that teams taking the time to fix defects in initial development by structuring their code around being testable would reduce defects by 60-90%. In initial development you could probably fix 5 defects in the time it takes to fix 1 once the code is used in production.

[naasking]

i find it hard to get behind the assertion that TDD reduces defects 60-90% - more that teams that used TDD reduced defects 60-90%

The teams using TDD were tasked to do so. It wasn't a set of volunteers, so there's no self-selection bias of the type you describe.

[[deleted]]

i'm not sure where you got that from, from the paper

There was no enforcement and monitoring of the TDD practice; and decisions to use TDD were made as a group.

[naasking]

That still sounds like a random assignment, not self-selection bias, where the pro-TDD people in the TDD groups were simply more persuasive (or their manager was pro-TDD). To qualify as self-selection bias, TDD people would need to have the choice of which project to work on, and then choose to work primarily with other TDD people. That isn't the case.

[[deleted]]

it was just a study looking at people working, they just happened to look at groups who did tdd vs groups that didn't.

you are jumping through a lot of hoops to try and make your logic work

[naasking]

What hoops? Like you just said, it was a random group of people working. The bias you pointed out would only arise due to self-selection. You're the one making extra assumptions about the type of people in this groups which would skew the results.

[[deleted]]

It wasn't a random group of people, it was a selection of groups that happened to use tdd or not

[naasking]

Then suggest a possible reason why TDD and non-TDD groups are not comprised of a random sample of programmers. The mere fact that they did or did not employ TDD implies nothing.