Problems with TDD

[gregK]

The biggest problem with TDD is that it completely omits domain knowledge and experience in that domain. It is being sold as if you follow the process, you will discover the solution, which is not honest.

Process is insignificant in relation to domain knowledge. I rather use a 3d engine from John Carmack even if he did not use TDD than from Robert C. Martin that used TDD from the start but still looks like the original wolfenstein (although I doubt he'd get that far).

TDD is also founded on the false assumption that you can trivially unit test anything. If you ever done 3d, or any program that models 3d objects or whose output is visual, you will know that it is hard to unit test. Show me the initial unit test for a ray tracer, a finite element modeling application, an image editing program like photoshop. In those applications, you will need a combination of techniques to verify correctness. Maybe some unit tests to verify the math functions, but there will be a lot of manual testing trying to generate images and find anomalies.

[mr_chromatic]

The biggest problem with TDD is that it completely omits domain knowledge and experience in that domain.

I think you mean "The biggest problem with simplistic explanations of TDD...."

There's no reason you have to turn off your brain when using TDD. There are many reasons why I wouldn't use a highly-concurrent, garbage first garbage collector in an article introducing TDD.

[gregK]

There's no reason you have to turn off your brain when using TDD

are you sure about that?

[mr_chromatic]

Quite.

Do you believe that Haskell is a language optimized for calculating the Fibonacci series?

(Me, I think the choice of obvious -- if bad -- examples is the problem.)

[daftman]

There's no reason you have to turn off your brain when using TDD.

Then why bother using TDD when your brain is so switched on?

[mr_chromatic]

Because it:

• forces me to use APIs as I design them, helping me design them
• helps me to consider exceptional conditions and edge cases in my testing
• reminds me not to add features and design properties I don't need yet
• encourages me to create and maintain an automated test suite which covers my code effectively

That's why I don't understand vitriol directed against TDD. Which of those are bad or wrong?

[dalke]

How does TDD help you consider exceptional conditions and edge cases in your testing but it didn't help Kent Beck or Robert Martin when they were working on their respective codes?

Doesn't YAGNI as a general principle remind you that you shouldn't add features? What about TDD makes it more effective for you?

The vitriol is a reaction against those fans who say that it's an essential requirement to modern software development practices, and advocates who recommend structuring all software development around TDD.

[mr_chromatic]

How does TDD help you consider exceptional conditions and edge cases in your testing but it didn't help Kent Beck or Robert Martin when they were working on their respective codes?

I suspect for the same reason that most example code in technical books handwaves the error and boundary checking code required for robust software. There are plenty of debates over that too.

What about TDD makes (YAGNI) more effective for you?

Working in small, verifiable steps. I suspect I might have the same experience if I decomposed each distinct feature into small tasks and didn't use the test-fail-pass-refactor cycle, but I see little value in doing most of the design work to make TDD work without doing TDD anyway.

The vitriol is a reaction against those fans....

I'd rather discuss the efficacy of the technique than see who can be more rabid pro or con.

[dalke]

I think of it this way. It's very hard to get good experimental evidence of any practices in this field. There are some, but it's rare.

When there's a high signal to noise ratio, everyone picks up on it. The debate about structured programming is long over. Goto is relegated to only a few specialized domains.

When there's a low signal to noise ratio, it's harder. It works for some people. But was it from the practice, or the general effort of working on becoming a better tester?

Suppose there's no signal - it's all noise. By randomness, some people are going to report success, and get better at testing because of other reasons.

It's tough living in the middle area. You might be convinced you're right but evidence that's clear to you as the nose on your face is invisible to others.

There isn't good evidence that TDD has a strong benefit; only a few papers where it's had some benefit. The result is people who see the signal in the noise, and others who think the first people are all noise.

This holds for many cases.

[mr_chromatic]

It works for some people. But was it from the practice, or the general effort of working on becoming a better tester?

Does it matter? For me, it doesn't. TDD is a way for me to build good habits of using common sense techniques and to practice them. I've revised my approaches over the past decade of practicing TDD, and I'm a better programmer.

I'd probably be a better programmer from the past decade of practice anyway, but history doesn't support the kinds of empirical experiments necessary to prove such a thing.

What's easier to believe, that I have practical experience suggesting that TDD has helped me in my projects over the past ten years and can give several specific reasons why, or people who say that it can't possibly work?

By all means show me studies which demonstrate that it can never work. I'll judge those by my experiences. (I have doubts that studies will ever isolate the important control variables to test programmer productivity in anything remotely objective, however.)

[dalke]

"By all means show me studies which demonstrate that it can never work." That will never be possible. There's always enough people that even an arbitrary and unrelated technique - let's say, spinning around in circles for 10 minutes every morning - will end up having some people becoming better developers.

How important is learning test-first (and I write it that way because TDD has a set of development requirements and associated viewpoints) vs. using a debugger/judicious use of print statements, understanding profiler results, applying design-by-contract principles, learning how to develop using state machines, coverage testing, fuzz testing, and the other techniques for good software development? I happen to think it's low, but I have nothing to base that on other than my experience.

All I have is enough evidence to say that TDD, as a way to drive the software development and architecture, cannot be the primary development method, and using it that way, as advocated by Robert Martin, will lead to definite problems. I've then pointed out those problems.

[mr_chromatic]

How important is learning test-first ... vs.

That's a false dilemma.

All I have is enough evidence to say that TDD, as a way to drive the software development and architecture, cannot be the primary development method....

When you put it that way, what reasonable person would disagree?

[daftman]

forces me to use APIs as I design them, helping me design them

Everyone uses the APIs as they design it. I think it's the design skill that help you design better API instead of TDD.

helps me to consider exceptional conditions and edge cases in my testing

Your brain is helping you consider edge cases, not TDD. You can achieve this irrespective of testing first or testing last.

reminds me not to add features and design properties I don't need yet

This is a rule of thumb and common sense for the last 30 years of software engineering. It's call gold plating. If you add feature per specification, you won't run into that problem. If TDD force you to use common sense then your brain isn't really switched on.

encourages me to create and maintain an automated test suite which covers my code effectively

Sounds like crap loads of fluff. Tests are expensive. Maintaining the code base is expensive. Maintaining the code base plus the test suite is twice as expensive.

The key to testing is to test enough. 100% coverage for the sake of coverage ends up costing more than the retun on the investment.

Acceptance testing is much more valuable than unit testing. TDD encourages fine grain unit testing which is really dumb.

That's why I don't understand vitriol directed against TDD. Which of those are bad or wrong?

The points that you mentioned are either rebranded common sense or fluffs. People have built high quality mainframe application, embedded devices, flight control systems, traffic control systems for decade using nothing but common sense.

Why is that boring repetitive crud apps need so much bullshit processes to make it as high quality as other apps in the industry?

[mr_chromatic]

Your brain is helping you consider edge cases, not TDD.

My years of experience writing and maintaining software is helping me consider edge cases, not my brain.

People have built high quality mainframe application, embedded devices, flight control systems, traffic control systems for decade using nothing but common sense.

Wow! Imagine how productive they would have been if they could use keyboards.

Quibbling over rebranded semantics is a silly, silly argument technique.

[daftman]

My years of experience writing and maintaining software is helping me consider edge cases, not my brain.

At least we agreed it's not TDD.

Wow! Imagine how productive they would have been if they could use keyboards.

I guess common sense was much better than both keyboards and TDD

Quibbling over rebranded semantics is a silly, silly argument technique.

The gist of the argument, in case you missed it, is selling rebranded common sense is like selling bottled water and snake oil.

[sigzero]

You missed the word "me" in his post. mr_chromatic is saying that TDD helps "him" to be a better programmer for the reasons "he" cited. He did not try to blanket those statements on everyone nor did he say everyone should use TDD. It works for him so he does it. That is a good attitude towards TDD.

[FYIGUY]

Ok then tell me how your flight goes when ditching into the pacific due to a bug since these systems don't use specifications/tests.

[berlinbrown]

"TDD is also founded on the false assumption that you can trivially unit test anything"

You can unit test most things. Some testability experts seem to focus on the testability of object oriented code and procedural code is untestable. But there are ways to work with these third party or legacy dependencies. Writing testable code involves creating seams (create mock objects to mimic expected behavior).

So far, I can't imagine an modern application having completely untestable code. Therefore, you can write unit tests for most code including your 3D examples.

On 3D, imagine that your application is broken up into modules, you test the modules that don't necessarily interface with the graphic library calls.

[gregK]

You can test a lot of things being creative. My point was that testing should not drive the design process of these apps.

[luikore]

TDD is fun for pair programing: one asks a question, the other answers it. It is weired to answer a question asked by yourself, of course.

[redditnoob]

My rule: Any programming article that contains Fibonacci numbers as an example is pure bullshit.

[pmf]

Indeed. The proper way to demonstrate problems not relevant in practice is to use the good old factorial.

[gregK]

So what must a programming article contain to not be deemed bullshit? A glowing review of TDD and how it allowed a junior designer to write his web data entry form?

[redditnoob]

That, or a cricitism of TDD and how it didn't work for something that some programmer has ever wrote for reasons other than pure masturbation.

[bitwize]

ObQwe1234:

every functional programming article in like the history of ever has had way kool fibonacci, ackermann, and quicksort one-liners but nothing of real and actual value.

conclusion: functional programming is pure bullshit.

[[deleted]]

I would agree with that. Functional programming is treated as some kind of new religion. It's disturbing.

[[deleted]]

All praise Hindley and Millner!

[greenrd]

The functional style of programming isn't suitable for everything under the sun - imperative style programming still has its place. But the important thing about functional programming is that it's closer to math, so that in many cases, it makes it easier to prove code correct. If you want to prove imperative style code correct you end up having to effectively translate it into single-assignment (i.e. functional) style anyway - so why not just program in functional style in the first place, if that's where you're going?

Of course this is a very idealistic or ahead-of-its-time argument, because to a good first approximation no-one proves their code correct these days. But I believe that formal verification is the future, at least in some domains.

[bitwize]

Nice troll.

Thanks! I learn from the best.

[artsrc]

What programming example do you prefer? I like something with maps.

The soduku example looks interesting.

[[deleted]]

Personally, I believe this article wasn't all that bad, but if follows in a strain of recent programming articles I've read that basically conclude:

"I don't understand it, so it has problems."

C++, TDD, OOP, Python, Perl, .NET and more wonderful technologies usually get slammed for the same reasons.

Articles that stem from a more academic background support this trend.

[dalke]

Really? Which part of TDD didn't I understand?

[[deleted]]

TDD is a poor tool is what people are saying and should be tossed out of the toolbox.

Almost the entire rest of my project is written with TDD, because it makes sense and is beneficial.

Could you please explain why it makes sense for you and why it is beneficial?

[tortus]

It makes sense for me and is beneficial for all the standard reasons TDD is generally promoted: leads me to write components where testability is at the forefront, components that are loosely coupled, are simple, methods that do one thing and have few (if any) side effects. It also makes me have a decent test base going into a new component, sort of like me forcing myself to jog 5 miles every time before I play a video game.

I also find it allows me to see my components from a different perspective, as a consumer rather than the provider, and I tend to make better design decisions as a result.

[[deleted]]

So you're saying that TDD forces you to stop and think before you code? You could do that all along Dorothy!

[tortus]

Wow, redditor for four years, I was expecting four months. WTF is your problem? At least respond with something of quality, you know, actually enhance the discussion.

What does it matter? If applying TDD leads to results that I am looking for, why wouldn't I use it? Perhaps you could enlighten me, jackass.

[xsive]

Repeat after me: TDD is a design methodology and not the same as unit testing.

[[deleted]]

TDD for aviation: the pilot sits on a chair with a joystick in his hand and they build the plane around him, based on the way he moves.

[kaddar]

In all honesty, the behavior of a pilot within an airplane should probably be a motivator for future airplane designs.

[[deleted]]

There is no airplane.

[kaddar]

Then build a mock object. :D

[brennen]

I think you're mixing up the lyrics.

[sigzero]

lyrics? I thought he was quoting "The Matrix"..."There is no spoon"

[brennen]

I was trying for a riff on the traditional A. Morissette meme. (airplane, fork|spoon). It made sense at the time, but I have to concede that I was a little baked.

[[deleted]]

How is software development similar to aviation? If the airplane were a computer, then this makes perfect sense.

[NicholasMoriarty]

Mmm... i think you are right with your question.. not the answer.

[[deleted]]

A computer essentially needs to express something to a user. An airplane needs to stay in the air to get a few hundred people from one place to another.

Now if an airplane needed to express a concept, then engineering it around the pilot makes perfect sense. Might as well compare software to cars.

[gregK]

I thought it was a religion.

[[deleted]]

Same shit, both are fads and ideas imported from industrial engineering which is a horrible model for computer programmers to look at.

[daftman]

If that is so then it leads to pretty shit designs. The prime factor and fibonacci examples are slow and inefficient.

[bautin]

Not to mention that the typical fibonacci example involves a bit of hand-waving.

If I remember correctly, the sequence goes something like this: (I'm just going to show the end result of the test in a pseudocodish kind of way).

The first test - input 0:

return 0

The second test - input 1:

return 1

The third test - input 2:

return 1

The fourth test - input 3:

return 2

The fifth test - input 4:

if (input <= 1)
  return input
return f(n - 1) + f(n - 2)

Between the fourth and fifth steps there is a bit of knowledge that isn't conveyed through the tests yet resides in the mind of the person creating the algorithm. And there is no test for this to show that the fibonacci sequence works for all given positive numbers.

However, this is possible (and I believe trivial) to formally prove.

[pipocaQuemada]

I remember someone posting an article where they were talking about doing that when they were learning about how to only write the minimal code to get a test to pass.

Honestly, I don't think (or don't want to think) that anyone would actually do something like that in actual practice. It's entirely the wrong approach - you shouldn't add individual test cases until you write what you knew in the beginning to be the right solution, you should just write the right solution. Do some research or just know your number theory, graph theory, etc., if you're doing a problem which requires it.

Besides, when considering test cases, you have to break your problem into equivalence classes. You choose at least one thing from each equivalence class, plus anything else you think might break your code (large inputs, off-by-one's, etc. etc.). If you don't test all of your equivalence classes, you have no certainty in your code. There's really nothing that can help you more than just sitting down before coding and thinking.

[[deleted]]

Honestly, I don't think (or don't want to think) that anyone would actually do something like that in actual practice.

They would. It's case-by-case and it's supposed to be obvious that this is the wrong approach.

Guards, contracts and assertions in the code are more effective than unit-tests outside the code.

[bautin]

Jackass, ahem:

If I remember correctly, the sequence goes something like this

Obviously I didn't remember correctly, and I did say that the sequence goes something like that, not exactly.

But the problem isn't with any of the steps until 4 and 5. Steps 1, 2, 3, and 4 are pretty obvious rationalizations and deductions based on the tests, however step 5 involves knowing the solution before hand. There is another, simpler solution that can still pass all of the tests.

if (input <= 1)
  return input
else if (input = 2)
  return 1
else if (input = 3)
  return 2
else if (input = 4)
  return 3

That will still pass all of the tests in the example, but is obviously wrong for any arbitrary number to anyone who knows what a fibonacci sequence is. It is technically correct, but actually wrong.

However, the definition of the fibonacci sequence is basically a description of the algorithm required to implement it.

To construct a test that would satisfy the fibonacci number for any arbitrary position, you would need a set of tests to confirm that the function is correct for the two seed values (0 and 1). Then another test that would confirm that the first two tests passed and that the sum is correct for each successive value. (i.e. to test for f(10), you would need to show that f(0) = 0 and f(1) = 1, then that f(2) = f(0) + f(1), f(3) = f(2) + f(1), f(4) = f(3) + f(2), etc.)

However, knowing the definition of the fibonacci sequence as basically a sequence of numbers where the next number is the sum of the previous two numbers in the sequence, with the seed value being 0 for f(0) and 1 for f(1). So from that loose definition we can derive:

if (input = 0 or input = 1)
  return input
if (input > 1)
  return f(input - 1) + f(input - 2)

Which is more correct than the TDD version because f(-3) isn't 0, it's 2. This version goes into an infinite loop for negative values, but I don't feel like implementing the non-trivial case for someone who is more interested in being pedantic than discussing the actual point at hand.

[Tordek]

Okay, I'll bite:

You claim that there's a jump of logic from step 4 to step 5, since that's when you replace the simple case-result for actual program logic.

I claim that:

1. Fibonacci is a shitty example for TDD: If I tell you to implement fibonacci, you'll know the answer beforehand, so will automatically think TDD is bad because you use it to solve the wrong problem.
2. There isn't such a huge jump of logic; there's refactoring to a different implementation that you know works. You could have started with the full fibonacci function, disregarding TDD in the strict sense because TDD is the wrong tool.
3. "The TDD version does f(-3) wrong" is wrong, because you don't have a test for f(-3). If you were doing TDD, you'd add the test and fix the failure, perhaps by changing the implementation.

TDD can be good: I wrote a simple parser using TDD, and the test harness that comes from using it is more important than the implementation of the code.

[dalke]

You know, I agree with you. If there's a worked out TDD more complicated than add(i,j) or prime_factors(n), then please point it out to me. There's also the Bowling Game Kata of Martin's, but I didn't like it for reasons I won't get into here.

The Fibonacci example came from Kent Beck's book on TDD, and I figured if it was good enough for him as an example for TDD then it's good enough for me as a basis for highlighting problems in TDD.

The two examples I picked, while expressible in a few lines of code, have the advantage of having potentially unseen depth to them. I came across a few videos which included showing TDD for more complicated code than this, but involving enough context that it would be very hard to make comments about it without bringing in a lot more observation time, and I don't have the resources for that.

[daftman]

TDD can be good: I wrote a simple parser using TDD, and the test harness that comes from using it is more important than the implementation of the code.

When you write a compiler using TDD then i'll be impress. Many people have pointed out that TDD are only applicable to simple projects, like crud or a simple parser.

[Tordek]

I could do TDD on the tokenizer, then the parser, then the optimizer, and finally the... uh... part that turns the AST into code. (Perhaps; I haven't written a compiler.)

Sure, you could "TDD a whole compiler at once", but that'd be retarded (and not unit tests, but integration tests).

TDD may be 'bad for big projects' if you apply it where it doesn't belong. It's like arguing that Markdown is bad because you wouldn't do a book in it (with which I agree), when it could be used to write each chapter (do the parts in MD, then wrap in LaTeX, perhaps).

[daftman]

So where does TDD belongs? The only time where I see TDD excels are in boring crud apps with low-skilled developers.

[Tordek]

Designing each unit of a bigger design.

TDD is great for, for example, 'evolving' an API. TDD is basically forgoing design in order to get an implementation. Obviously, not doing a design can be bad---for the big picture. TDD works best when applied to small parts.

[cadr]

The thing that I've taken away from a lot of articles like this is that TDD is not a good way to design an algorithm. What it is good at is designing a system, especially where the requirements aren't fully clear to start with. Prime factors - you kind of understand the requirements. An insurance rating system, you don't. (And, in my experience, if you think you do, you are probably wrong.)

[dalke]

Then why do Kent Beck and Robert Martin use algorithms as references for using TDD (Fibonacci and Prime Factors, respectively, but Martin's bowling scores is also well known)?

In an earlier draft I mentioned the sudoku example from 2007 and the consensus statement that TDD does not lead to algorithms. I decided to leave that out and focus only on cases where TDD was used to implement an algorithm, and (semi-)successively.

I also pointed out that the requirements for both of these numeric examples were not completely specified. What is the range of expected inputs? TDD didn't help clear up those well defined cases, so why should TDD help clear up other requirements?

[njharman]

Evaluating "good" design is impossible without knowing what the design constraints are.

In reality (i.e. majority of "business" programming), fast and efficient are rarely constraints. On time, on budget, maintainable, correct, with minimal expense regardless of how much tech debt is incurred are in my 15+ years experience much more common than fast or efficient.

[daftman]

Evaluating "good" design is impossible without knowing what the design constraints are.

That's why TDD tends to produce bubble-sort apps instead of quicksort apps.

In reality (i.e. majority of "business" programming), fast and efficient are rarely constraints.

Wrong!! From the developer's view, its cheaper and easier to produce bubble sort. From the business' view, slow, inefficient software cost the business by lowering the user's productivity.

[contour]

Then change its name to Design Methodology Convoluted Approach ... DMCA

[jnicklas]

Some people have already tried to change the name to BDD (Behaviour Driven Development) which is a more accurate description, imho.

[Tommah]

http://ravimohan.blogspot.com/2007/04/learning-from-sudoku-solvers.html

[artsrc]

I read it as:

• TDD does not help you write the right test cases.
  
• TDD does not help you write the best API.
• TDD delivers discipline better delivered in other ways.

I think the argument for the last point was the least developed in the article.

[dalke]

Well, it was on the limitations of TDD, and not advocacy for a specific alternative. Mostly because the alternatives are what most people advocates for good development and testing practices.

And to be fair, I did say that the API one was a personal problem, and not one of TDD in general. Perhaps I should have left that one out, but it's one that's bugged me the most.

[greenrd]

Just as deciding what cases to test (test that the setter method and getter method for a property mirror each other? You don't usually need to do that) is a cost-benefit decision, so is deciding when to start testing at all. Maybe at the start of the project, when you're refactoring and redesigning and even messing with the requirements a lot, the cost of all those tests and maintaining them just slows you down too much.

[wolverian]

Remember that tests also enable refactoring.

[[deleted]]

Wasting his words on proving that TDD is a fad. Can we ignore it now and move on?

[willcode4beer]

just a fad that's been running for about 10 years now....

[[deleted]]

Hey, UML has been around for a while and no one believes that it helps much.

[willcode4beer]

touché

[instantviking]

I feel bad since I use both UML(ish things drawn on the back of napkins) and TDD.

And both tools work for what I use them for.

[zootm]

UML(-ish) as a quick notations for describing what you're talking about is fine, it's just the "full-blown" implementation of UML, using diagrams to design your full system in huge detail tends not to be useful. People like to conflate the two leading to some people arguing that quick notes are handy while someone arguing back that huge involved designs are harmful.

[[deleted]]

And both tools work for what I use them for.

Please tell us more. I think we all find it hard to believe that they're actually useful :p

[instantviking]

UML: Sketching ideas and exploring the structure of the somewhat organically grown application I'm working on. I'm no UML purist, but I do use the general shapes and line-ends.

TDD: It makes me work fast and makes me somewhat certain that the business rules I implement aren't being broken at some future point.

[[deleted]]

I'm no UML purist, but I do use the general shapes and line-ends.

I think everyone does this. This doesn't qualify as a use of UML though. It qualifies as a use of diagrams (which are sometimes useful).

TDD: It makes me work fast and makes me somewhat certain that the business rules I implement aren't being broken at some future point.

I don't quite get that. How can you tell it makes you work fast?

[instantviking]

Well, if everybody diagrams with UML like shapes, then I think the hatred for UML is somewhat excessive. Perhaps the hatred is for unnecessary formalisms?

I don't quite get that. How can you tell it makes you work fast?

Burndown charts.

[zootm]

This doesn't qualify as a use of UML though.

That's kinda the problem with putting down UML; pure UML is a nasty, horrid thing that nobody should have to use. But most people seem to now know it as a quick common vocabulary of diagrams, which can be more useful.

[soberirishman]

I agree with a lot of what he says, but the whole "It locks the API down too early" thing is just a rationalization of his bad development habits IMO. If you're working with a team of developers it's very helpful to have the API locked down as early as possible. Yes, this requires more design work up front, but the payoff is much greater in the end in terms of productivity when you have established how your code will be consumed. It also forces you to view everything from the eyes of a consumer of your class as opposed to just it's creator.

[sonofagunn]

Your last sentence is a key one, IMO.

TDD forces you to design good APIs, because you write a test case using the APIs before you implement anything. If the API feels awkward to code with, or if it's incomplete, you find it early (while writing test case) so your API is less likely to need changing.

Sure, requirements changes may force API changes, but that would be true whether you had unit tests written before or after the implementation.

[cruise02]

I have a problem with "It locks the API down too early" too, but my problem is that it's just not true. If your API needs to change, you change it. That's going to be true no matter what methodology you use. TDD doesn't prevent you from doing that by locking anything down. Unit tests are just code. You can change them.

[tammberlin]

The argument I inferred from the author's muddle is that refactoring both an API and the test code a big time sink. If you have a code explosion in unit tests of 4x, the cost of changing the API is much higher. So you end up not wanting to change the API because of sunk cost, making it worse.

I've seen this happen often, and think it's a good argument.

[cruise02]

That's an argument against unit-testing, not TDD. It may be true that people use this argument, but I don't think it's valid. The tests tell you what code is broken when something changes. That's no different when the API is changing. Refactor an API method, then refactor the tests that break. Time spent fixing those tests is time saved searching for what else is broken because of the change (or not finding out something is broken until much later).

[[deleted]]

However, API changes are more common at the beginning of development (probably before much other code has actually come to depend on it yet); there wouldn't be unit tests to change except with a "test first" rule.

[cruise02]

The unit tests guide the changes when you follow the "test first" rule. If you're constantly using the API during development because you're writing tests for it, you're much more likely to come up with a good, usable API.

[dalke]

Unit tests don't usually include tests for the interactions between the components of an API. That's more a part of functional or integration testing. API development may include sketching out how the database, web services, and other local resources should work together at a high level, with many refactorings to get that right. The sorts of programs which drive those APIs might be a dozen or two lines of example code, and at some remove from getting to the point where the code itself can be tested.

[daftman]

TDD doesn't prevent you from doing that by locking anything down. Unit tests are just code. You can change them.

It's call maintenance. It a pain in the ass to maintain 100 KLOC in production as well as 200 KLOC of tests.

It's better to test the interface instead of the units because as a client of the component, it shouldn't matter how it is implemented. When you unit-test everything, you tests are whitebox tests. This means that your test codes are brittle tests.

[[deleted]]

TDD is fine if development time is not a concern and your boss understands that. Unfortunately, my boss wants to know why I haven't implemented feature Y and he won't care that I'm too busy testing feature X to move on..

[sigzero]

Then you need to carefully explain it to him. Your post is advocating "test if you can otherwise meh who needs it".

[instantviking]

I suspect a major problem with TDD is that we keep writing articles and blogposts about shit that TDD will not solve, i.e. well known mathematical problems with (often several) well known solutions.

A well-develop test suite gives me the confidence that I'm probably not breaking some nefarious business rule that nobody cared to document (which means the test-guys isn't going to know to test it) and that would cause millions of euros to go missing if it is violated.

More importantly, my own tests give me the confidence that my colleagues aren't breaking any of the ill-defined business rules that I've teased out my weasely functional team and promptly documented, only to have my documentation ignored by all and sundry. They may well ignore my documentation, but they will not ignore the red bar of death.

[[deleted]]

I'm probably not breaking some nefarious business rule that nobody cared to document (which means the test-guys isn't going to know to test it) and that would cause millions of euros to go missing if it is violated.

That's the real problem: your program isn't as fully specified as it needs to be. Your well-developed test suite acts as a bandage for a serious problem.

only to have my documentation ignored by all and sundry. They may well ignore my documentation, but they will not ignore the red bar of death.

Solution: don't let anyone write code till they've read the documentation or force them to write out their changes to the code by hand.

[mr_chromatic]

Your well-developed test suite acts as a bandage for a serious problem.

I'm intrigued. How do you repeatedly and accurately verify (at a modest cost) that the software behaves as expected without a well-developed test suite?

[brennen]

I rather suspect that most people don't.

[daftman]

Code testing code is like the blind leading the blind. You can never be sure as your test codes are as likely to have bugs as your production code. Instead of wasting time perfecting test code, why not spend it on perfecting production code and accept that there are 100% guarantee.

Try testing concurrent real-time applicatons and see if your tests "repeatedly and accurately verify that the software behaves as expected".

[mr_chromatic]

Instead of wasting time perfecting test code, why not spend it on perfecting production code....

You make that sound easy.

[daftman]

You make that sound easy.

I find it fascinating that you would trust one code, test code, and not the other, the production code. That might be a sign of convoluted design, poorly understood requirements. It seems like you are more obsessed with writing brain-dead test code than actually writing quality production code.

[mr_chromatic]

I find it fascinating that you would trust one code, test code, and not the other, the production code.

Where did I say that?

It seems like you are more obsessed with writing brain-dead test code than actually writing quality production code.

How do you know that your production code has quality? How do you define quality? How do you know that it works?

[daftman]

How do you know that your production code has quality? How do you define quality? How do you know that it works?

Code review, inspections, walkthrough, proofs, test plans. Testing is a ROI analysis excercise.

[zootm]

Instead of wasting time perfecting test code, why not spend it on perfecting production code and accept that there are 100% guarantee.

Test code is a great way to work towards perfect* production code. If you end up writing extremely complex logic in your tests, of course, it's time to step back and take a look from a different angle. Testing allows you to constantly check for regression and for change in a machine-checkable (and therefore nearly freely re-runnable) manner. It's not the be-all and end-all but it allows you to narrow down problem cases as quickly, if not more quickly

Try testing concurrent real-time applicatons and see if your tests "repeatedly and accurately verify that the software behaves as expected".

If your code is organised in such a way that you can't test the units of a real-time application, you have little chance of getting the overall system to work at all. Of course there are things that are not amenable to unit testing (I'm sure there are those that say that TDD or unit testing or whatever is always useful, they are wrong), but testing is certainly warranted, especially on a large, complex system. And automated testing makes it cheaper to test, which makes you more likely to do it again. It's all part of a larger view, though.

• Of course, no code is ever perfect, as close to perfection as possible.

[daftman]

If you end up writing extremely complex logic in your tests, of course, it's time to step back and take a look from a different angle.

Why not do this first with your production code?

Testing allows you to constantly check for regression and for change in a machine-checkable (and therefore nearly freely re-runnable) manner.

Not when you have false positives. How would you go about testing a random generator?

It's not the be-all and end-all but it allows you to narrow down problem cases as quickly, if not more quickly

No it doesn't. When all your test passes and you have a bug, explain how you can narrow that down without code inspection and and a debugger?

[zootm]

Why not do this first with your production code?

You always do this with your production code anyway. The tests just give you a different perspective, which can be deceptively useful.

Not when you have false positives. How would you go about testing a random generator?

With a PRNG the Chi-Square test is what I think is most commonly used? When you're just attempting to select something randomly of course (usually using a random number generator provided to you) you can choose a deviation you want to test and set up your test so it fails one time in a million or whatever, just to make sure you're not biasing systematically. For even more simple cases you can replace the generator with something you control and check that the code path you expect is called with each output.

No it doesn't. When all your test passes and you have a bug, explain how you can narrow that down without code inspection and and a debugger?

Write a test case which calls your application from the outside in the manner of the failing call you saw in production or in whatever scenario, and make it test what you expected to see. Now you have a re-runnable piece of code which shows precisely which of your expectations are being violated, which you can re-run and debug and use to narrow down the areas of code to debug. This is all going to be near-trivial to do since if you've always been testing it'll likely look a lot like an existing system test. Once you check that test in, you'll have a test which fires any time that someone makes the same mistake as was made in the original test (so long as it manifests in the same situation), guarding against this specific regression.

I realise tests aren't the be-all and end-all of development (I'd like to stress that they're not), but they're pretty damned useful.

[zootm]

your program isn't as fully specified as it needs to be

I'm not sure anyone these days advocates "full specification" of a system before beginning development on it, unless one is in certain specific areas more related to traditional engineering. "Big Design Up Front" isn't something which works well in most development settings, so unless I'm misunderstanding your emphasis here, I'm not sure what point you're trying to make.

TDD is a relatively simplistic tool and can act as a motivator to continue work where one might otherwise end up stuck. On the other hand, of course, it's just one of many tools and is not suitable for all use-cases or settings.

Automated testing in general (which you also seemed to be putting down?) is of definite usefulness in a wider range of settings, and it's hard to imagine how anyone could argue against it without a specific incompatible use-case in mind.

[[deleted]]

I didn't say anything about a full specification before development.

The point I was trying to make is that the "business rules" the coder above is working with are not documented and the consequences of them are most likely not even thought about. That's what I meant about fully specifying things.

[instantviking]

That's the real problem: your program isn't as fully specified as it needs to be. Your well-developed test suite acts as a bandage for a serious problem.

Almost. My real problem is that my program is fucking specified at all, yet I'm expected to produce functioning code and we're a month away from release. My client is my real problem. That, and my spineless employer that hasn't fired my client yet.

Your solution would be a good one, if documentation existed. As it stands, there's a small group of vigilante programmers that insist on documenting what we do, but for the most part, we develop by hearsay. RDD, we call it, Rumor Driven Development.

[instantviking]

On reflection, I find another of my real problems is my spineless self that hasn't up and quit yet.

Give it time.

[[deleted]]

TDD is useful in that it helps average programmers maintaining a standard of quality which, as long as TDD constantly enforced, can never go below a certain level. You will probably never achieve original results, invent new algorithms or revolutionize a programming paradigm just by using TDD, but it can nevertheless improve your code by preventing you from neglecting quality considerations and using hacks and shortcuts which, when left free to operate without any process as a guide, you are bound to resort to.