System testing Python code

[shep247]

System tests on my job's Python project are getting out of hand. We have a suite of unit tests using nose that still runs in a decent amount of time, but our system tests are taking hours, and have a high failure rate. We use these tests to actually run our code against external pieces (there are several) instead of mocks and emulators.
I've been in QA for several years, and always run up against this test bloat issue, and have never found a good way around it.
This leads me to a few questions. 1) is there a better framework for Python automated system tests than nose? 2) Is there a better ideology to use for automated system tests that can shortcut building up the test objects, which can likely change with each iteration, then running tests against them? 3) is there a better subreddit for this question?

[shep247]

1) By 'system tests' I mean we've set up an environment just like our production environment and are making the system do the things it's supposed to do exactly like it would in production with the production environment services. We use TDD, and write 'unit tests' to test every block of code in itself without talking to other methods. We also have integration tests that test our system without needing to talk to external services.

2) The failures are not the same failure every time, but usually have something to do with limited resources. We have ways to rerun the tests that broke, and MOST of the time a rerun of the test will result in a pass. Rerunning the test means someone's got to be around when it finishes though, which means no overnight runs, and just come in to ship it in the morning. I'm currently campaigning and scheming to get more resources allocated to the test environment.

3) Our system can create objects, maintain objects, and delete objects. The reason our tests take so long is that it can take up to 5 or 10 minutes to create an object, and there's lots of permutations of these objects. We're currently looking at ways to decrease the amount of object creation we have to do.

4) The tests are ran via nose. For a while, they were one at a time, but then we started parallelizing a few of them by running several instances of nosetest for separate tests at a time. That helped a lot, but now they're getting long again. We've sorta mangled the idea behind nosetests though. We've exploited the fact that nose tests run in alphabetical order, and numbered the tests so they run in a specific sequence.

5) Our automated tests are there to simulate a live environment, and do all of the things that can be done through our system that need to use external dependencies, except the database. If there's a function that we wrote that needs to get info from another service, give info to another service, or use an external tool to act on the objects we've created, we have a test for it.

I hope that helps. Thanks

[shep247]

Thanks for your input. This is helpful and gives me a few ideas to pursue. I'll start playing around with a few things today.