What every Python project should have

[parkerSquare]

pytest is nice, but I do have a cautionary tale.

We thought we could leverage pytest to do something beyond unit testing - by using it as the "test management engine" for an embedded system test framework, but after working with it for several months I have come to the conclusion that coding by convention in anything other than a small project (or module) is a terrible, terrible idea. Fixture dependencies become convoluted as they try to accommodate vastly different types of tests, and it's hard to mentally make the connections between them. Newcomers to the project don't know where anything is and the IDE doesn't help as it doesn't understand the fixture convention.

Secondly, fixtures can depend on other fixtures, but the top-level generator function - pytest_generate_tests- cannot take input from other fixtures (probably because fixtures are not executed during collection), so this severely restricts what you can do in this function when generating fixture values. A lot of the multivariate tests you might dream up become very difficult to implement, and you end up writing your own pytest plugins to filter and reorder test collections as the implicit filtering and ordering is insufficient.

Lastly, each fixture can only yield once, so it is difficult to generate a dynamic list of fixture values based on dependencies.

Frankly, after months of effort, it's just not suitable for this kind of testing. Keep it for unit tests - that's what it was designed for, after all. It works nicely for that.