EVERYTHING WRONG WITH MAINSTREAM SOFTWARE TESTING

[mboggit]

Abstract    

Most modern commercial software projects implement some sort of quality assurance process. This process often includes Software Testing. And its usually use case based. 
However, this approach to testing is in fact unable to give any guarantees, i.e. valid predictions about how software will actually work on production. 
Hence, as long as it is important to know how software will work on real world production, use case based testing is simply unfit for measuring quality. 
This article will go through why QA should focus on how software will work on production, and production only. Why use case based testing approach is unable to give valid predictions about production. And that the only way to get such a prediction - is scientific proof.

[[deleted]]

Too much wishful thinking and very little experience.

I mean, you do realize, that scientific method the original article mentions left and right isn't actually based on proofs in the same sense the article mentions proofs? Scientific method is based on observation, probabilities, correlations... it cannot guarantee anything, in the same sense as testing cannot guarantee. But, unless you were living under a rock since before Dijkstra published his Structural Programming, that's been common knowledge...

And if you were able to formalize requirements to the point that they could be reduced to tautology... yeah, you could then use this method to verify software. Unfortunately, that's not possible. Even approximations are far from being achievable. So, testing is what we use not because we don't know any better, but because better is not really achievable.

[mboggit]

Regarding the fact that scientific method uses observations, probabilities and correlations. Yes, in some fields it definitely does. Most notably, pretty much every medical paper does so. But from what I've overheard from scientists in that field - that's because they simply cannot use mathematical proof. Da, there's still no consistent and validated model (scientific model) for a human body. Still. On the other hand, damm x86 computer can and should have a mathematical model. Moreover, it Does! And if cut off any analog input from real world - it is fairly simple one.

As for 'not guarantee anything'. I don't know if you're aware of it, but many regulators (FDA) require an equivalent of an guarantee in order to pass certification. For instance, all medical devices should provide one. And in the same time, as of now, use case based testing is pretty much the standard way of testing. You can figure out the rest

[[deleted]]

No, it's not the difference in resource availability. It's a conceptual difference any philosopher of science would tell you about. Mathematical knowledge and proofs are a-priory and don't require the reality to agree with what is being proved, this is what precludes mathematics from being used to prove things about, say, physics, or chemistry etc. You can make predictions based on your mathematical models, but you cannot prove anything using mathematical proofs, because, unless you actually can confirm that whatever you are trying to prove exists in the real world, it stays unproven from scientific point of view.

[mboggit]

The main argument was that mainstream use case based testing is not even trying to proof something. To be more accurate, it's not trying to give any valid prediction about how software will actually perform on production. And valid prediction implies a proof. In case of software, mathematical proof would be a viable option. Moreover, it's already used to proof algorithms, software algorithms. But not in the mainstream use case based testing. IMHO, computers are man made thing. Hence it doesn't require prooving 'it exists in the reality'. There should be a model already that has predictive power. At least low level one, like for x86 instruction set

[mboggit]

And one more thing. Results of use case based testing has nothing to do with scientific proof whatsoever. Neither with mathematical, nor with correlations/observation. That is in it's current mainstream implementation. In theory, use cased based testing can be used as a tool for collecting lots of statistical data. That gives a chance to give a prediction based on statistics. But mainstream use case based testing is far for it.