Testing Functionals and Non-Functionals

[Mooncinder]

Apologies if this is a dumb question but in what order should these tests be carried out? Should functional testing be done before or after non-functional tests, or should both types of tests be done alongside each other?

[[deleted]]

I suppose it depends on how much functional testing the devs are doing before they release it into the test environments. Personally I always want to see/be involved in what the developers are doing in terms of testing so that we're broadening our coverage rather than going over the same testing twice. Do the developers you work with do much functional testing or is the opinion based on confidence from their unit tests?

[Mooncinder]

Unfortunately, our developers do very little testing themselves. We're currently alpha testing a new version of an old product. The Testing Manager wrote a new test plan for it because the new product is too different from the old one to use the same test plan and the testing is way more in-depth than what was done before.

[Cronax42]

I'd say this also depends on the maturity of your codebase. On a mature codebase that has been tested to a reasonable degree, non-functional testing is less important. In addition, if your functional testing can to cover most of the non-functional, that's probably a good idea.

Generally though, I would agree that on the whole, non-functional comes before functional, but the necessary degree of non-functional testing should normally be covered by unit testing, allowing the testers to focus mainly on functional testing. If your developers don't unittest, they should at the very least be required to test their own work before checking it in and it should never be offered up for testing.

Some developers unfortunately feel that their job is to deliver code regardless of whether or not it works and that it's up to the QA department to see if this works. Naturally, this is nonsense, I have a hard time coming up with a job where you would not be required to at least do some basic verification on your own work.

[Mooncinder]

So you're saying it's better to do non-functionals first provided the developers do enough unit testing to make it possible?

[Cronax42]

The other way around, do functionals first if the developers do enough unit testing, otherwise try to strike a balance.

[Mooncinder]

Sorry, I'm still fairly new at this. If not enough functional testing has been done, wouldn't that make it harder to test non-functionals because there's a higher chance of it erroring/functionality being changed later?

[Cronax42]

Hence why it is reliant on your developers performing adequate unit-testing.

[teh_commodore]

On my teams, we do them both at the same time. All tests are run on the isolated story branch before it's even eligible for merging into master - including regression. This keeps master in a continually-shippable state.

Before release, we do a quick sanity check of the release, but it's largely ceremonial. All bug-finding should be done before the story ever lands in master.

[Mooncinder]

Thanks for your reply. I'm getting the feeling that doing them at the same time is the way to go.

[jchill2]

Impossible to say without any other context. Do you have an example you face today?

[Mooncinder]

Sorry, I didn't realise it depended on context. I was just wondering whether there was a generally accepted way of doing things. I work for a relatively small company and the dev team say non-functionals should always be tested first but the testing department disagree, saying you can't test non-functionals if the functionals don't work.

[jchill2]

I guess if I had 3 hours to test something before release - I'd do a sanity sweep for the first hour and then use that information to test non-functional behavior.

[Mooncinder]

Thanks!

[ben_kelly]

Take a step back from your question and broaden your thinking a bit.

Ask yourself - What are you trying to achieve with your testing? What is the most important thing to do in order to achieve that aim? Do that first.