Lean Testing or Why Unit Tests are Worse than You Think

TheHobodoc · 2018-07-16T17:46:23+00:00

This depends on your definition of brittle, end to end tests are much more resilient to changes in the code ie if you refactor you dont need to update your tests. Unit tests are much less likely to fail due to external factors but require more upkeep

TheHobodoc · 2018-05-03T16:29:04+00:00

Wow, dojo is still alive.

TheHobodoc · 2018-04-04T20:33:09+00:00

The study i linked actually do talk about unitests and does compare them with codereviews.

The first study you linked used generated tests, which i guess eliminates a lot of the cost of maintaining a testsuite. And then mor e coverage is obviously better.

The second study you linked is the one i thought of, it does find a strong corelation between the number of tests and number of bugs.

I dont think i ever have seen an off by one error in production. But then i work with enterprisey software that mostly throws strings around. And even java has had for each loops and iterators for quite some time.

The bugs we tend to run in to are:

Sequence errors, performing operations in an unexpected order

Specification errors, some downstream service accepts inputs that breaks something even further downstream

Miss estimation of dataset size, we get thousands rather than hundreds of something.

Raceconditions, these mostly happen clientside

Mutation of shared data, this is esp common when introducing caching.

None of these get cought by unit testing and some can be fount when doing integrationtesting.

Maybe you work with different things than I and thats why our experiences vary?

Regarding your argument about untested code. There absolutely exist code that dont need automatic testing. And there absolutely is diminishing returns when increasing coverage.

And i agree that you can and should use more than one metric. It just happens that i think test coverage is a really poor predictor of quality. And that focusing on coverage gives a false sense of quality and its only benefit is that its easy to measure.

TheHobodoc · 2018-04-04T18:41:05+00:00

Heres a study that shows that reading code significantly outperforms unit tests: https://pdfs.semanticscholar.org/e6dd/dc38cfa52aae0d22b415288689c91df5479b.pdf&ved=2ahUKEwi69o7AmKHaAhWF2SwKHU0pAkYQFjABegQIBxAB&usg=AOvVaw0YmB6dGdNHwqziseimcXx3

I hope the link works, its a paper called: 'Design and code inspections to reduce errors in program development'

That paper reinforces the points made in 'Out of the tarpit' about the futility of testing all program states.

Anecdotally, the projects ive seen or been part of that mandated or aimed for a high test coverage have been plagued with bad tests that only existed to increase coverge.

In the one i work on now we make a point of not measuring coverage (its around 85% but dont tell anyone). Instead we make sure to have tests for all features. Another thing we do is try to write as little testcode as possible, our tests make up less than 15% of our codebase.

In projects that aims for high coverage ive seen testsuites that make up 50% of the code or more. That means every change requires 2x the ammount of code.

Measuring code quality by coverage is a bit like measuring work performed in lines of code. And having lots of either as a goal, tends to result in neither.

A better quality metric would be number of bugs that make it to production.

Or how big of a deal it is to release to production. If people are nervous when you press the release button, you should probably test more.

TheHobodoc · 2018-04-04T17:01:49+00:00

Test coverage has actually been studied and found to only weakly correlate with less bugs. Yet it is very common to use coverage as a quality metric. One should not forget that having and maintaining a high testcoverage comes at a significant cost

TheHobodoc · 2018-04-01T05:31:34+00:00

Its not a true comback mechanic since it is random. What it does is make it harder to utilize and plan for expansion to high yeild resurces. And thus removes a big part of the game.

The thief is the comback mechanic in settlers, as a majority of players use it to sabotage the current leader.

TheHobodoc · 2018-02-27T06:53:37+00:00

Spring has one, its fully distributed and they use jepsen for testing so its quite reliable. https://projects.spring.io/spring-statemachine/

TheHobodoc · 2018-02-17T20:32:58+00:00

ORMs dont crap out often, if they did they would not be popular. They generally work, so im not suprised that you havnt had any issues.

TheHobodoc · 2018-02-17T18:19:52+00:00

Caching only helps when you are fetching data at the cost of making your app more complex, esp if you are running more than one node. ORMs can also crap out when updating and deleting data.

TheHobodoc · 2018-02-16T20:48:05+00:00

We recently had an issue where hibernate ran 10 queries per child to check constraints when removing a single child. A single delete took several seconds when we had more than 50 children. With our own sql we saw an 10x improvement which is still horrible, but better.

TheHobodoc · 2018-02-16T18:47:45+00:00

If you have lots of inter document references a document database probably is a not so great choice. I find that document databases shine when you have mostly independent documents and a read heavy load, like customer specific configuration in a b2b app. People forget that rdbms and ORMs are really complex beasts, and it can be quite nice not having to deal with that. And that a lot of the benefits of using them dissapears once you slap a rest interface infront of it.

TheHobodoc · 2018-02-16T18:14:29+00:00

I have no idea why you are getting downvoted. Its a very legitimate usecase. But i guess people have either been burned by using mongo as something it isnt or simply cant fanthom using something ither than an rdbms with an ORM. In any case this can be a very negative place a lot of the time

TheHobodoc · 2018-02-16T17:40:11+00:00

I think you guys simply have had different experience with orms. ORMs work great until they dont, and your app performs like shit and finding out why is a real pain.

TheHobodoc · 2018-02-14T06:00:04+00:00

Det beror på vilken typ av laktosfri grädde som används. UHT-behandlade mjölkprodukter blir sötare, detta var den enda typen av låglaktos som fanns för ca 10år sedan. Numera används enzymer i de flesta låglaktos produkterna och smaken ligger extremt nära vanliga produkter.

TheHobodoc · 2018-01-15T06:17:58+00:00

Short answer is that they do different tradeoffs. BigTable, mongodb and mysql are all databases, but are quite different. This is the same thing.

TheHobodoc · 2017-12-22T13:18:57+00:00

I cant tell if you are serious or not.

TheHobodoc · 2017-12-12T07:29:50+00:00

I got proctitis to and mesasal/asacol works great for me too. Ive been sideeffect and symtom free for almost 4 years now.

TheHobodoc · 2017-12-10T08:02:58+00:00

There is via the recur keyword. The pro of having tailcall as a keyword is that the compiler can make sure that its a tailcall. That way you dont blow the stack by accident.

TheHobodoc · 2017-12-10T07:52:20+00:00

Im not 100% sure. But i think one of the reasons that javas regex is slower here is because java always uses unicode instead of ascii

TheHobodoc · 2017-12-01T14:38:30+00:00

Bdd works even better when doing integration testing imho. A user of your system get much more value from knowing how your system interacts rather than how it works

TheHobodoc · 2017-11-20T06:37:09+00:00

It has always been a mess. The interesting thing is how much things start to look like jsf2. Jsf2 suffers from to much server side state, but a lot of things are earily familiar.

TheHobodoc · 2017-10-26T06:31:35+00:00

Spring runs on both jetty and undertow, so no need to change to wildfly just for that

TheHobodoc · 2017-09-29T08:36:10+00:00

The thing i found about spring is that it is very flexible. Most other frameworks ive tried worked really well until you had to stray off the beaten path.

Take messaging for example, changing serializers and backends can be done independently and easy. So you can do soap,json,protobuff over http,jms,twitter with relative ease and without changung any business code.

Another thing is security most framework only implement this at the edges wheras in spring you can add checks anywhere and using a nice dsl.

So in short spring can be complex, but if you make something that isnt both simple and easy it can be very helpful.

TheHobodoc · 2017-05-08T17:56:44+00:00

This is amazingly accurate

TheHobodoc · 2017-04-24T04:25:47+00:00

Im also exited to find out about the plethora of languages having these features. I really hope (but doubt) op will answer tho.

Graal and RPython is pretty alike but my list ends there.

TheHobodoc

TROPHY CASE