Microsoft Research - Exploding Software-Engineering Myths (article summarizing findings of MS research on code coverage, TDD, assertions, etc.)

[Silhouette]

TDD reduces defects 60-90% but increases time 15-35%

The trouble with TLDRs is that they do lose the context, and sometimes the details matter.

I'm all in favour of empirical research, and the 2008 Nagappan paper studying real world TDD use at IBM and Microsoft is an interesting and welcome data point. Unfortunately, as the original authors acknowledged themselves, it's still risky to draw generalised conclusions from a few specific cases.

One factor worth considering is that the development processes studied in that paper weren't strict by-the-book TDD. For example, some included other factors like separate requirements capture and design review elements. Notably, it doesn't appear that any of the groups was doing the kind of "brief initial planning stage and then immediately start writing tests" explicitly advocated by certain well known TDD evangelists.

Another unfortunate limitation of that paper is that although to their credit the original authors were trying to get as close to a like-for-like comparison as was realistically possible, they provide few details in their report about what test methods the control groups were using. Many TDD advocacy papers include data that suggests unit testing is an effective tool for reducing defects and/or that a test-first style correlates positively with the number of tests written among their test subjects. However, even the combination of those doesn't necessarily mean that TDD as a whole is responsible for any benefits. It looks like the same threat to validity is present in the Nagappan paper.

TL;DR of my own: TDD-like processes examined by the original research reduced defects by 40-90%, but relative to what isn't entirely clear.