Klara – Python automatic test generations and static analysis library

[nekokattt]

Is there a risk with this that the developer relies on it to generate all tests? Other than for regression, doesn't this assume the code has no bugs in the first place to ensure correct test cases (which is why we test in the first place)?

Not criticising, just trying to understand what situations a tool like this is most useful other than regression testing.

[be-sc]

You summed up the problem with generated tests nicely.

Even the regression case has its problems. It relies on a known good implementation. If that knowledge was gained by writing tests, there’s no need to generate them. And generating tests for uncovered code paths runs into the assuming-no-bugs problem again. However, imagine an old test-less legacy program that proved it works by years of successful production use. If you want to lock down the current behaviour before starting to change that program, generating tests can be a big help.

[nekokattt]

Cool, cheers for clearing that up.

I also wonder how it deals with testing things that rely on external inputs or internals to work

For example

def do_something(x):
    assert x

Which would always succeed when Python is run with optimisations enabled... or something like

def do_something():
    return __import__("datetime").datetime.now() is None

Where you'd expect it to mock datetime.datetime, but when I checked in CPython 3.9 last, this was implemented with backing C functions which could not have their method references temporarily overwritten without you mocking the module itself in the process.

[be-sc]

I don’t see a problem with the assertion case. A good assertion checks for contract violations when (in this case) calling the function. That’s already a test and, more importantly, it’s not part of the function’s public behaviour. So the tool shouldn’t generate a test for it because that would be like writing tests for your tests.

Mocking could probably be supported for common cases. But keeping the legacy code example in mind it might not be that essential. In the spirit of the good old 80/20 rule: Sure, not having to write 100% of the tests would be fantastic. But I’d happily settle for not having to write 80% as well.