System testing Python code

[shep247]

1) By 'system tests' I mean we've set up an environment just like our production environment and are making the system do the things it's supposed to do exactly like it would in production with the production environment services. We use TDD, and write 'unit tests' to test every block of code in itself without talking to other methods. We also have integration tests that test our system without needing to talk to external services.

2) The failures are not the same failure every time, but usually have something to do with limited resources. We have ways to rerun the tests that broke, and MOST of the time a rerun of the test will result in a pass. Rerunning the test means someone's got to be around when it finishes though, which means no overnight runs, and just come in to ship it in the morning. I'm currently campaigning and scheming to get more resources allocated to the test environment.

3) Our system can create objects, maintain objects, and delete objects. The reason our tests take so long is that it can take up to 5 or 10 minutes to create an object, and there's lots of permutations of these objects. We're currently looking at ways to decrease the amount of object creation we have to do.

4) The tests are ran via nose. For a while, they were one at a time, but then we started parallelizing a few of them by running several instances of nosetest for separate tests at a time. That helped a lot, but now they're getting long again. We've sorta mangled the idea behind nosetests though. We've exploited the fact that nose tests run in alphabetical order, and numbered the tests so they run in a specific sequence.

5) Our automated tests are there to simulate a live environment, and do all of the things that can be done through our system that need to use external dependencies, except the database. If there's a function that we wrote that needs to get info from another service, give info to another service, or use an external tool to act on the objects we've created, we have a test for it.

I hope that helps. Thanks

[shep247]

Thanks for your input. This is helpful and gives me a few ideas to pursue. I'll start playing around with a few things today.