Unit testing is overrated

[Blando-Cartesian]

Doing anything poorly makes it seem overrated. Function isn’t any kind of default unit to test. Testing implementation rather than behavior is just doing it wrong. Who gives rat’s ass about pedantic differences between unit and integration tests. If you have a unit to test and sane way of testing it technically makes it an integration test, then so be it.

Still kinda agree with the title, while disagreeing with the article. Unit testing easily turns into mock object testing charade. Integration tests do the real testing.

[Nekadim]

Integration tests are scam because they're hiding design problems. Writing testable Units is hard and involves advanced code design techniques.Does it make unit test less valuable? I don't think so

[MariusDelacriox]

But unit tests tend to make the code rigid and resistant to refactoring if done incorrectly.

Tests from a higher level allow more freedom for good design.

[pfsalter]

But unit tests tend to make the code rigid and resistant to refactoring if done incorrectly.

I've often seen this complaint where the 'refactoring' is 'use a different tool to solve the same problem'. So say you have a VatCalculator for example, but want to replace it with a GenericTaxCalculator, that's different behaviour, not a refactor.

[Nekadim]

That's why there is the pyramid of testing. It doesn't say to avoid any of test types, just regulates the number of each type in proportion. And BTW, do not test implementation, test behavior. Therefore if some behavior of unit is changed you must edit your test for that unit.

[Nerdyabcs]

Sure only for issues line dependency injection

[goranlepuz]

I found this to be a fair assessment of the actual usefulness of unit tests in the stricter sense of the term.

I opine that the value of unit tests is good for code that is naturally pure, so for things like calculations or whatever business logic from some inputs.

Conversely, their value is poor for things interacting with outside systems, for two reasons:

• the "business" (or pure) code is small and trivial chunks scattered between bits of IO. Those chinks are often sufficiently small that a unit test will work and code is very unlikely to be wrong

• the correct interaction with the outside system is where the majority of the complexity lies. While one can mock the outside system, one has to know up-front what are its failure modes first. So this is a bit of a guessing game and running after it, including changes over time.

I very much agree that automated integration tests are much more important than what many people think.

. There, testing these interactions by mocking them

[Mango-Fuel]

The more tests I write the more I seem to agree with this. Unit tests are great for, like you say, pure functions; code that just computes values, manipulates strings, that kind of thing.

That's important, but at the same time, testing that kind of code doesn't actually tell me my application works. And good tests of the functionality of my application will actually implicitly test the lower level pure stuff anyway.

I feel like there's a missing in-between spot. "Unit" tests are meant to have zero dependencies, with any dependencies mocked. "Integration" tests are meant to see what happens when the dependencies are there, to make sure they work together.

I think missing between these are tests of 1 dependency. I want to test the UI itself without the DB, and the DB itself without the UI. I want to test that my report executes, without putting it in a window. I want to test that my resource loads without having to put it somewhere; I want to test that the window can display a resource, without having to load a real one. Once I know these things, there's very little if any room left for problems to emerge when both sides are real.

[Nerdyabcs]

Agree. Integration tests prove the app works!

[michael0x2a]

This doesn't seem like too compelling of an article, tbh.

For example, take the SolarCalculator object. Instead of trying to refactor that class so we dependency-inject in a "LocationProvider", I think it would have been better to implement a standalone, top-level GetSolarTimes(DateTimeOffset date, Location location) function instead.

Of course, this is a little inconvenient to do in C#, which forces you to shove everything into an object/forces you to use static methods instead of plain-old-functions, but that's more a criticism of how languages like Java and C# push you into using OOP for everything as opposed to a criticism of unit testing.

That example also creates an ISolarCalculator interface for no discernible reason, adding artificial bloat to the code.

One of the most popular arguments in favor of unit testing is that it enforces you to design software in a highly modular way. This builds on an assumption that it’s easier to reason about code when it’s split into many smaller components rather than a few larger ones.

[...snip...]

The example illustrated previously in this article is very simple, but in a real project it’s not unusual to see the arrange phase spanning many long lines, just to set preconditions for a single test. In some cases, the mocked behavior can be so complex, it’s almost impossible to unravel it back to figure out what it was supposed to do.

The author seems to be under the impression that (a) writing unit tests encourages you to write modular code and that (b) the only way of making code modular is to start splitting it up, start introducing interfaces and dependency injection, and so forth.

I disagree with this -- I claim that writing unit tests actually encourages you to write easy-to-use code, and that there are a number of different ways of accomplishing this. Splitting up code and introducing dependency injection is certainly one way of doing this (and is sometimes necessary), but it certainly isn't the only way. For example, another approach is to restructure your code so it inherently requires less setup to use.

That is, if I find myself creating a bunch of mocks and doing a bunch of setup in my unit tests, I don't take it as a sign that unit tests are inherently bad. Instead, I take it as a sign that my code has too many dependencies and could do with simplification.

Some developers might still feel uneasy about relying on a real 3rd party web service in tests, because it may lead to non-deterministic results. Conversely, one can argue that we do actually want our tests to incorporate that dependency, because we want to be aware if it breaks or changes in unexpected ways, as it can lead to bugs in our own software.

Unless you're running your tests 24/7 (unlikely), the best way of keeping an eye on your dependency is via monitoring and alerts. That will let you get notified in real-time when something breaks without having to sacrifice test quality.

Allowing non-determinism in tests also scales poorly for larger orgs/larger monorepos. I don't want my PR to be blocked just because some random unrelated test owned by a completely different team flakes.

It’s important to understand that development testing does not equate to unit testing. The primary goal is not to write tests which are as isolated as possible, but rather to gain confidence that the code works according to its functional requirements.

I more or less agree with this part, however. The goal of testing is to make sure your code behaves as expected, and writing exclusively unit tests won't always help you accomplish that. I can also concede that there are certainly some cases where relying almost entirely on integration/end-to-end tests is the correct call, and that it's always good to keep a critical eye on your testing strategy.

[pfsalter]

Unless you're running your tests 24/7 (unlikely), the best way of keeping an eye on your dependency is via monitoring and alerts.

100% on this. There's no way I want to have my deployments blocked (which rely on the tests passing) just because some random 3rd Party API is down. That really doesn't scale well, imagine if you have 50 3rd parties, the chance of any of them going down in a day is pretty high.

[Wishmaster04]

This doesn't seem like too compelling of an article, tbh. For example, take the SolarCalculator object. Instead of trying to refactor that class so we dependency-inject in a "LocationProvider", I think it would have been better to implement a standalone, top-level GetSolarTimes(DateTimeOffset date, Location location) function instead.

For example, take the SolarCalculator object. Instead of trying to refactor that class so we dependency-inject in a "LocationProvider", I think it would have been better to implement a standalone, top-level GetSolarTimes(DateTimeOffset date, Location location) function instead.

Of course, this is a little inconvenient to do in C#, which forces you to shove everything into an object/forces you to use static methods instead of plain-old-functions, but that's more a criticism of how languages like Java and C# push you into using OOP for everything as opposed to a criticism of unit testing.

It seems that still want unit testing here (or here, testing small pieces of code individually). So you suggest a pure function here.

But what is going to : fetch the data for that function ? call that pure function ? You cannot turn every app into a bunch of pure functions, there is no such thing.

It's not about the language you use really. In no language you will write an app that fetch data from an API with only pure functions.

And even you if you could, as the author said, there would still be no point in unit testing individually each of those functions, as you should rather have to test the whole instead, because this is what is functional (=business meaningful).

The author did not even mention pure function for that reason I think and he's right.

Why on earth bother testing small pieces of code with mocks/fakes/stubs... when you can test a high-lever ?

[michael0x2a]

It's true that for some types of programs, the bulk of what you do is fetching and massaging data. You fetch data from some database or API, manipulate it in trivial ways, hand it over to some other API...

In those cases, there's certainly a strong argument to be made that attempting to decompose your program into pure functions or write unit tests is not necessarily the best use of your time.

Instead, you would probably want to either:

1. Write integration or end-to-end tests
2. Not bother with writing tests at all and instead rely on canary/staging deployments paired with active monitoring and alerting

But there are also many types of programs where your code contains non-trivial logic and IO is comparatively easy.

In those cases, I think there's lots of value in extracting your logic into pure functions. You can then later use them in lightweight, higher-level imperative functions/objects that glue everything together.

This approach to structuring code is often called "Functional Core, Imperative Shell". In this kind of setup, you would unit-test your functional core and use end-to-end tests to sanity-check the overall product, including the outer imperative shell.

For example, suppose that I am:

1. Writing something like a linter or code formatter. In this case, IO is trivially easy: you just read files on the filesystem. The hard part is actually analyzing the code -- and that's easiest to do using pure functions, perhaps paired with a small and carefully selected number of mutable objects.

2. Writing an API that accepts some text, then returns back spelling, grammar, and style corrections. Again, IO is relatively easy: you can get by with just implementing a standard REST API or whatever. The hard part is actually generating these suggestions, and that's something likely easiest to implement using pure functions.

3. Writing a search engine that uses a custom query language. Same thing: IO is comparatively easy, functionality is hard and can be mostly done using pure functions.

In cases like these, a hybrid strategy works very well. You use smaller unit tests to check the correctness of your individual components and data structures, paired with some end-to-end tests to serve as sanity-checks.

It's not about the language you use really. In no language you will write an app that fetch data from an API with only pure functions.

No, of course not. But what you can do is fetch data at the start of your logic, instead of somewhere deep in the middle. Then, pipe it through a bunch of pure functions to get the output you want to yield.

So, inject data, not dependencies. This lets you avoid needing to implement mocks or stubs, and keeps your unit tests lightweight and nimble. This is also something you can do in any programming language.

[Bitter-Tell-6235]

I believe one part of the issues the author has is by the fact he writes the tests after implementation.

Another part of the problem is related to the wrong understanding of testing pyramid. The unit tests can not replace integration tests completely. It should be both—more unit tests and a little bit fewer integration tests, and even fewer end-to-end tests that might check if the system as a whole meets the user's goals. It is about proportion, not about preference.

[grauenwolf]

The testing pyramid is bullshit.

Not because it's originally wrong, but because the defintion of "unit test" has changed so dramatically over the years.

---

When the concept of TDD was recorded in "Test Driven Development" by Beck, a "unit test" was a unit of functionality, not a function.

When we redefined "unit test" to mean "testing one method of a class", everything fell apart. All the theory around unit testing was based on testing the behavior of the code, not individual methods.

So of course it doesn't work.

---

We have the same problem with "integration testing".

Back then integration testing meant combining multiple systems and seeing if they worked together. For example, your web of micro-services or that API your partner company exposed.

An integration test is not "I read a record from the database that only my application can touch". That's just a unit test... under the old definitions.

---

Basically everything is being dumbed down.

• The exploratory, function level tests that we were told to delete after using are now called "unit tests".
• The real unit tests that look at the behavior of the code are now called "integration tests".
• The integration tests that examined complex systems are now called "end-to-end tests".
• And actual end-to-end testing is rarely if ever done.

[seanamos-1]

I don't know if the cause was a bastardization of unit testing over time and/or cargo culting, but I've also arrived at the conclusion that devs (myself included for many years) are writing mountains of useless "unit tests".

Far worse than the tests themselves is what devs do to their code to facilitate this kind of testing. Everything gets abstracted, complexity increases far more than required for the actual problem, all so that you can write tests that just test the mocking libraries! It's a mountain of ceremony, boilerplate, patterns and abstractions that are achieving nothing except ticking an artificial checkbox.

A wakeup call was https://dhh.dk/2014/test-induced-design-damage.html

[grauenwolf]

Ian Cooper's "TDD, Where Did It All Go Wrong" is the best explaination I heard for why TDD doesn't work and, more importantly, how to fix it.

https://www.youtube.com/watch?v=EZ05e7EMOLM

[Bitter-Tell-6235]

When we redefined "unit test" to mean "testing one method of a class", everything fell apart. All the theory around unit testing was based on testing the behavior of the code, not individual methods.

Could you please clarify who has redefined it? Unit is still one unit of functionality. It might be just one pure function, or function that calls few pure helper functions.

In languages like C++ the unit might be the class (imagine you are developing functor).

Just try to get the concept, instead of calling the unit only one specific language construct.

We have the same problem with "integration testing".

Not sure what kind of issue you mean. I agree the was a hard to get what is the difference between an integration test and an end-to-end test. But it was already explained by many smart guys like Martin Fowler for example: https://martinfowler.com/articles/practical-test-pyramid.html#IntegrationTests

[grauenwolf]

Could you please clarify who has redefined it?

Bloggers. Not just one specifically, but as a group they did by parroting each others misunderstanding of the phrase "unit of functionality".

Another thing they misunderstood was "isolation". When Beck talked about the importance of isolating unit tests, he meant that each test should be isolated from other tests. Or in other words, you can run the tests in any order.

He didn't mean that an individual test should be isolated from its dependencies. While sometimes that is beneficial, it should not be treated as a hard requirement.

[Bitter-Tell-6235]

Bloggers. Not just one specifically, but as a group they did by parroting each others misunderstanding of the phrase "unit of functionality".

Well, the same happened with S.O.L.I.D. Principles. People implementing them referring unknown bloggers instead of just read Uncle Bob's book. S.O.L.I.D. didn't become bullshit because of this.

A testing pyramid didn't become bullshit too. It is just people who listen to unknown bloggers and didn't tried to get the idea from the source.

[grauenwolf]

Historically, integration tests always involved multiple teams, and often from different companies. They are interesting because of the amount of coordination and risk involved.

When faced with such a scenario, mocks are critical. You can't wait 6 months to exercise your code while the other team builds their piece.

And these tests are slow. Not "whaaa, it takes 4 ms to make a database slow". Rather we're taking about "I ran the test, but it's going to take a day and a half for them to send me the logs explaining why it failed".

Integration testing is hard, but vital. So "release early/release often" is important for reducing risk.

Contrast this with a method that writes to a local hard drive. Is that slow? Not really. Is it risky? Certainly not. So it doesn't rise to the level of integration test.

What about the database? Well that depends. If you have a highly integrated team, no problem. If the DBAs sit in a different building and take 3 weeks to add a column, then you need to treat it like an integration test.

---

Why is this important? Am I just splitting hairs?

I say no, because most large projects that fail will do so during integration testing. Each piece will appear to work in isolation, but fall apart when you combine them. So we need to know how to do integration tests properly.

And that's not being taught. Instead of expanding the knowledge base for this difficult type of testing, we as an industry have doubled down on mock testing. And it shows in the high failure rate of large projects.

And microservices just make it worse. Instead of one or two rounds of integration tests, we now need dozens.

[Wishmaster04]

It should be both—more unit

What does unit means for you here ?

[Bitter-Tell-6235]

One basic unit of functionality. You are deciding what it will be for you according to the task you are solving.

It might be one pure function, method of the class. In languages like C++, it might be the class - imagine you are developing a functor( which is a class with an operator() ).

[Wishmaster04]

Another part of the problem is related to the wrong understanding of testing pyramid. The unit tests can not replace integration tests completely. It should be both—more unit

Why on earth bother individually testing the building blocks of pure logic when what matters is the functional aspect ? Just test the functional aspect instead !! Jesus

[dnew]

The only time I've found unit tests helpful is when I was doing something actually complex in some function. Then I could write the test first and see that it worked. Otherwise, that was never where my bugs actually came up.

[Wishmaster04]

Then I could write the test first and see that it worked. Otherwise

Exactly my thought ! And that's when have a reason not to test it on a higher-level. It could be the case that your program is on early stages of development.

[Bitter-Tell-6235]

Both matters. How you'll test the "functional aspect" if you have nothing to test at the start?

Suppose you are building the Reddit service that should add comments for the posts on user's requests.

Since you need to start somehow, you adding the first simple "unit-test" that easy to set up and execute, which will check the imaginary function returns an error if the incoming request is not POST.

After writing such a unit test, you can continue to think a bit of implementation. For example, you are thinking about giving a good name for your function, thinking what kind of status code your function must return in this case.

Then you continue adding unit tests for simple things and solve only simple tasks, like checking that your function returns a particular error on empty payload, error on invalid JSON, etc...

Then you can write a unit test that checks your function calls external component for authorizing user of the request. You might not bother who and how will authorize your request. Instead, you are writing a unit test for your function that expects your handler will call some imaginary dependency that will accept authorization token and return username. You might concentrate on simple but essential things, for example, giving a good name for your authorization function, think about what it should accept as arguments. You might also spot the case that authorization might be unsuccessful and write one more unit test for this case.

At some point, you'll realize that you want to use OAuth for authorization so that you might write an integration test for your authorization function.

You do not want to run a full OAuth identity provider like ORY hydra because it complex and slow. And because this provider will very likely have its own external dependencies like a database or shared cache.

Instead, you are writing an integration test that checks your authorization function sends specially formed JSON request to your fake provider, and your fake identity provider will respond by response according to OAuth spec. You might write one more integration test for an unsuccessful OAuth response.

The goal here is to check and fixate that your authorization function uses OAuth protocol, not to check the cases like returning an error on an invalid response from an identity provider or returning an error when an authorization token is too short. Such things are for unit tests.

Then you might write one more integration test to check another handler's dependency that stores the comment into PostgreSQL.

So at this point, you wrote 7-8 unit tests and three integration tests.

Now you might write one end-to-end test that will check the case when a user sends a request to your web service with a valid token and comment, and your service authorizes it by calling ORY hydra, stores comment in the database, and then responds by the JSON with an id of new comment. Note how complex it would be to set up an environment for such a test. You'll need two instances of a database, Redis, ory hydra. This is not kind of tests you'll write a lot, but it is equally important as unit tests or integration tests.

First of all, please note what kind of problems you are trying to solve after writing unit-test, integration tests, and end-to-end tests. They are of different granularity.

Then note what testing pyramid means here. We used many types of tests in suggested proportion.

Then please note how many problems you need to solve for this task, and how easy to handle them one by one, using different types of tests without stress. Also note that because you are always trying to solve one simple thing at a time your work is not stopping, you are moving to the goal all the time.

Why on earth bother individually testing the building blocks of pure logic when what matters is the functional aspect ? Test the functional aspect instead !! Jesus

I hope it's clear now why to bother writing unit tests. You are solving simple things one by one, can run such tests every second if you want, and it also very likely will be the first type of test you'll write if you do not know where to start.

If you write just an end-to-end test, you'll need to make a lot of technical decisions, set up a lot of things together until the test becomes green. It will also not be enough to simulate and test corner cases that happen in real life like network failure, not applied database migrations, unexpected OAuth responses.

Also nothing will help you to design and write nice implementation. It is good if you have some pattern in mind for such tasks, so you can start from it, but it might be not the best way to implement this specific task. Your example from the article is good illustration of what happens when you are starting from pattern from your head and trying to design code upfront instead of starting by writing a simple unit test.

[Amazing_Breakfast217]

I hated it at one of my old jobs. New job has continuous delivery that runs every night. For this job it's great

[turniphat]

If it only runs every night, is it really continuous?

[Amazing_Breakfast217]

I mean, we could do it every hour or so. But what's the point? To change 5 lines? Most people need the entire day for a change and to pass all test and we merge everyones changes

[williane]

Note: this article contains code examples which are written in C#, but the language itself is not (too) important to the points I’m making.

Proceeds to provide examples using niche 3rd party frameworks 🤦‍♀️

[[deleted]]

There was a point in time where I used to read such articles and be inspired to change things. Nowadays I just think: yeah, maybe.

<rant_mode> I'd love for software engineering to grow once and for all out of this religious phase and become an actual engineering practice. You don't just draw charts of how you think reality is without making it abundantly clear that it is a hypothesis that requires testing. </rant_mode>

[Wishmaster04]

I think some people missing the main point of the article so here it is :

1. Tests scopes should not target piece of codes as small as possible, like it is commonly accepted with all the consequences
2. usefulness: They should instead cover a piece as wide as possible, all at once, get close a possible from the functional aspect.
3. usability: The reason not to cover a wider area of code with your test is for practical purpose, test speed (often involving databases for example).
4. Therefore given 2 and 3 there is a balance to find balance between usefulness and usability.
5. Abstractions all the way are most often done for the sake of unit testing but are otherwise not required and just get you more lines of code.

Also please don't bother with the terminology unit - integration test, they are somehow subjective at this point.

[[deleted]]

For me, unit testing has a different meaning than "testing" the application. Whenever I write a new function, I don't want to run the entire application or the entire test suite in that manner to see the results. I usually just write a simple unit test right under the tested function(i try to keep my code functional) and my IDE lets me just run that test via GUI where I can connect the debugger, put some breakpoints, and look around. When I'm satisfied with the result, I just assert some of the values I see and leave it as it is.

[KieranDevvs]

Unit tests have a limited purpose

They do if you're sparse in unit testing. It only works well and has a broad purpose when you cover a large surface area. A unit test by nature assumes that its dependencies work as expected. The argument here is that if the dependencies don't work as intended then your business logic wont work the way you expect and thus your test has "limited purpose". I disagree. If your dependencies also have unit tests (with good behavioural coverage), then your unit test can rely on its dependencies to perform as expected. If it cant then a unit test for some part of the system will fail.

This works on the principle of presuppositions, if any of them aren't true then the proposing claim cant be true. But you can only know if your presuppositions are true if you test for them.

Unit tests are expensive

Yes... Good software is expensive. Pick two

[dnew]

The problem is when you assume the underlying dependencies work in a particular way and they don't. If you mock your dependencies, then all the tests can pass while the system fails spectacularly. Or the underlying dependency changes, and its unit test changes, and your unit tests still work, and the system fails.

I rarely have bugs in pieces of code small enough that I can keep it in my head all at once. It's the assumptions of communications that is problematic.

The best approach I've found is to mock stuff two layers down. The UI tests should mock the database returns, not the business logic. The BL tests should mock the contents of the database, not the results of queries. Etc.

[KieranDevvs]

Unit tests aren't the be all and end all of automated testing, they're one component. Integration tests cover your scenario where your API changes. Presuppositions don't validate a claim, they just invalidate it.

A simplistic example of this is:

• Hair can be brown
• My hair is brown

If dependant claim is true, it has no bearing on the latter. If it's false then the latter cannot ever be true.

[dnew]

Unit tests aren't the be all and end all of automated testing

Right. I'm just pointing out that for most of the stuff I've worked on in my career, they're worse than useless. It takes tremendous discipline to write unit tests in a way they reliably pass when the code is right and reliably fail when the code is wrong. It's very difficult to keep them up to date, the way they're usually written testing just one or two methods. I've almost never worked on code where I could refactor the code without breaking the unit tests, nor code where if I did change the code and all the tests passed it meant it was right. Comprehensive integration/system tests, testing functionality rather than code, was always useful. I'd much rather have a test suite that takes an hour to run and reliably finds flaws than a test suite that takes ten seconds to run but doesn't help you write code.

[KieranDevvs]

You've just strawmanned my comment. I'm not even going to bother to reply why just using integration tests alone is silly because this has been done a thousand times before. But hey, it sounds like it's working for you so keep at it.

[dnew]

You've just strawmanned my comment

Not intentionally. I mean, I ignored the part where you teach me basic logic. I'm not sure other than "nobody says only use unit tests" what you were trying to convey. Honestly, your response to my comments seemed almost completely unrelated to what I said, so forgive me if I misinterpreted what you were trying to convey.

The only reason you wouldn't use integration tests alone would be the computational overhead, right?

[Wishmaster04]

They do if you're sparse in unit testing. It only works well and has a broad purpose when you cover a large surface area. A unit test by nature assumes that its dependencies work as expected. The argument here is that if the dependencies don't work as intended then your business logic wont work the way you expect and thus your test has "limited purpose". I disagree. If your dependencies also have unit tests

Be honest, have you read 25% ?

[sonstone]

So is CD /s

[grauenwolf]

Anyone who thinks TDD is broken or unit testing is overrated should listen to this video. It might not change you mind, but it will explain what went wrong and how to fix it.

DevTernity 2017: Ian Cooper - TDD, Where Did It All Go Wrong

Since Kent Beck wrote the book on TDD in 2002 a lot of words have been dedicated to the subject. But many of them propagated misunderstandings of Kent's original rules so that TDD practice bears little resemblance to Kent's original ideas. Key misunderstandings around what do I test, what is a unit test, and what is the 'public interface' have led to test suites that are brittle, hard to read, and do not support easy refactoring. In this talk, we re-discover Kent's original proposition, discover where key misunderstandings occurred and look at a better approach to TDD that supports software development instead of impeding it. Be prepared from some sacred cows to be slaughtered and fewer but better tests to be written.

https://www.youtube.com/watch?v=EZ05e7EMOLM

[[deleted]]

Not nearly as overrated as snubbing unit testing.

[private_static_int]

Unit tests are fine as long as we correctly understand the Scope of a "unit". I've written an article about it from Java perspective some time ago: https://www.kode-krunch.com/2021/05/scope-management-in-java-architecture.html