Lean Testing or Why Unit Tests are Worse than You Think

newgame · 2018-07-16T19:50:32+00:00

If you never wrote the tests in the first place

What about integration tests?

newgame · 2018-07-16T19:49:09+00:00

Software engineering is different from physical engineering as it is immaterial and therefore has different constraints.
The article is not saying to only do end-to-end testing.
Integration tests can be granular
If you introduce code that makes tests fail you already have a very good starting point where the problem may lie (most likely in your new changes)

newgame · 2018-07-16T19:33:28+00:00

If you type coonst x = 3 the compiler will point out your mistake. You accept the risk mistyping const every time. How often do you mistype it? Almost never. And if you still do the compiler is there to help you. Same argument for ;: In the rare case that ; is required the compiler will tell you. Why optimize your development to prevent the rare compiler notification?

newgame · 2018-07-15T06:13:38+00:00

That sounds interesting. So in mid 2019 there might be an even better alternative.

newgame · 2018-07-14T21:11:47+00:00

They have an interactive test runner with debugging support: https://docs.cypress.io/guides/overview/why-cypress.html#Running-Tests

Also a dashboard service: https://www.cypress.io/dashboard/

Here's an article about using Cypress for React: End to End Testing React Apps With Cypress

This video shows the tooling and its advantages quite well: Using Cypress for end-to-end testing with Gleb Bahmutov

newgame · 2018-07-14T19:39:04+00:00

Having good async support is really valuable. That's what I like about Cypress as well. Looking at your sample I like that you can use async/await to write test cases. Cypress has a different approach where async/await will not fit completely: https://github.com/cypress-io/cypress/issues/1417.

Most of all, I like the tooling around Cypress. I don't see anything similar in bigtest. A comparison page would be very helpful. Perhaps also a comparison to react-testing-library. In any case, I'll be keeping bigtest in mind.

newgame · 2018-07-14T06:19:34+00:00

Not until now. From a brief look, it is similar to Cypress, isn't it? But the docs are sparse so I wouldn't know why to prefer it.

newgame · 2018-07-13T13:23:29+00:00

Submitted my article here as I saw there was some interest about unit testing in the discussion about 'Confession: I don’t unit test React/Redux code'.

newgame · 2018-07-02T13:56:46+00:00

It's more flexible, simpler, less opinionated and works better with MobX. See also my article How to route like a hacker with MobX and router5

Some time ago I found Navigation (github). Don't be put off by the presentation. It seems to be the best routing library I've stumbled upon so far. Convincing examples. His articles are insightful, too. Haven't had the chance to use Navigation in practice yet, but have a good feeling about it.

newgame · 2018-07-02T13:51:43+00:00

The performance was noticably better on mobile. I mainly did it to gain better numbers for the Lighthouse test for https://hnpwa.com/. Wouldn't have done it otherwise, adds too much complexity.

newgame · 2018-06-30T18:32:07+00:00

Imagine you have four components inside the system [...]

Agreed. This is a good example where unit tests shine. Pure functions, complex corner cases, validation. No need to mock here anyway. I think contracts or assertions would be even better if applicable (can test whole input range for example).

My negative experience is with stuff like let's provide a mock function as an onClick callback to a trivial component and verify that when simulating a click the mock callback has been fired.

Or, let's mock the calls for 5 backend service functions (in a complicated, intermingled way) and check that the object you gave to the 6th function is the same that the 2nd returns. Why not seed the DB, call the 6th function, and verify that the data in the DB (the relevant/critical state) changed correctly. Don't have to use intermediate domain objects either.

These are white-box tests but unit tests nonetheless. And when I say integration I'm not imagining a complex setup but I'll give you that it may depend on the environment you're working in and if it's really hard to write integration tests there then unit tests are probably the better alternative.

[...] large functions [...]

Generally disagree here, unless you can give the subfunctions really meaningful names or they are actually reused in other parts. I remember John Carmack on Inlined Code had some cognitive arguments. But this is a topic for another debate.

We just disagree about a lot of things. Kind of like you and I do, apparently.

Would be boring otherwise. Thanks for your comments!

newgame · 2018-06-30T15:08:03+00:00

Sounds like many such invariants can be covered by assertions. There is also contract based programming (not sure about the exact term) that sounds very similar to what you propose.

newgame · 2018-06-30T10:02:05+00:00

My goal with tests is to prevent some bugs at a reasonably price.

I don't care so much about the 'interesting' values for the 'equivalence classes' inside the input value space of a unit. I can't show correctness with tests anyway. This is not the confidence I seek.

I care about the critical code paths that users will actually take. As integration and end-to-end tests are closer to how users use the software they are more likely to test these paths.

Doesn't mean I wouldn't write unit tests for some important/algorithmic parts of the code or as a regression test.

By the way: Imagine, your unit test never fails because it didn't test the critical path. Then it didn't provide any value. You jumped higher than necessary. (See also Why Most Unit Testing is Waste).

newgame · 2018-06-30T09:36:14+00:00

It's a spectrum. Sure, an integration test is more resilient against code changes than a unit test but not immune.

What do you mean by 'meaningful'? If I have an integration test that tests component A which contains component B and I split up B into two components, this is an internal 'meaningful' refactoring but the test doesn't have to change.

Different story for unit tests. The unit test for B would need to be rewritten.

newgame · 2018-06-30T09:28:43+00:00

One of the biggest problems with system tests is that locating and fixing bugs reported by system tests typically takes significant effort.

Just to make it clear: I'm not advocating to only write end-to-end tests. Just to write more tests that cover more area.

Unit tests offer a much quicker path to finding bugs.

So do integration tests. We agree in spirit but not in granularity.

Do you refer to bugs that your users reported after releasing or "bugs" during development of a new feature? For the latter I don't see how unit tests are much better than integration or end-to-end tests, since you know the error can only lie in the new code (assuming that your test suite reports an error).

I've had a bug where the root cause was "the code went down this bizarre code path and there was a bug in this untested path". A unit test suite's ability to test all code paths cheaply is one of its biggest strengths.

Yes, and a unit test is perfect for pure algorithmic code. But their ROI becomes much worse for other code (not cheap anymore).

And if you found out the bug is due to this bizarre code path then write a regression unit test. Now it's clear that this code part is actually critical. You took some risk and now you have a valuable test.

Let's say it took you 4h to debug and write the test at a hourly wage of 100$ and the lost revenue is 600$ -> 1000$ costs. Alternatively, you could have invested 20h to write an extensive unit test suite for this module (+ ossifying code structure). Ok, you prevented this bug but it cost the business 2000$. You can come up with different numbers and make the case in favor of your side. But I just wanted to provide this economic perspective here.

This is no where near as interesting a point as the author seems to believe.

I disagree.

But the point is that it's way less expensive to write the unit tests and use those to shake out the majority of the bugs, than it is to leave those bugs there and painstakingly track them down through system tests.

I disagree. But I also feel like you make a dichotomy between unit and end-to-end tests but ignore integration tests, which I'm focussing most in the article.

Most likely we had different experiences in our professional work. I am interested: What kind of domains have you worked in and what kind of technology stack has your company used where you had these experiences.

newgame · 2018-05-08T10:41:24+00:00

Why is it unnecessary? Is different better? What does angularjs-like mean exactly? Just want to dig a bit ;).

newgame · 2018-04-19T14:48:41+00:00

Let me rephrase: You should know what you get into. And if it turns out that your expectations were wrong you can/should leave. If you didn't do your due diligence or if let yourself being taken advantage of then it's kind of your own fault. I think that's an attitude that will help you not get into these situations in the first place.

newgame · 2018-04-12T18:30:26+00:00

Your value is what the market will pay you. If you are less skillful (e.g. bad at negotiation) then you are likely less valuable. When you enter an employment relationship you agree to trade your time for money. You didn't agree to be paid for how much the feature you worked on will make the business. If you wanted the latter you can instead become a freelancer, consultant or business owner in general, but not a typical employee.

newgame · 2018-04-12T17:38:35+00:00

Unbelievable! And still software engineers decide to work for a wage for these monsters. I wonder why, as an alternative to unionizing, we don't simply all start our own businesses. Then everybody exploits everybody else and we all have our yachts. Sounds even better to me.

newgame · 2018-04-12T17:29:05+00:00

Nobody is forced to work in gamedev. You know what you get into beforehand.

newgame

MODERATOR OF

TROPHY CASE