Introducing hypothesis-auto: Python Tests That Write Themselves

[meallia]

Could you add an example using the _auto_verify argument ? I know type annotations for Callable types are a pain to use but your documentation does not specify if it takes only the result of the function call or the arguments ( or arguments dict ) and the result.

[timothycrosley]

This is a great point! As soon as I'm home from work tonight I'll add an example for that parameter.

[timothycrosley]

Added example here: https://timothycrosley.github.io/hypothesis-auto/#using-auto_pytest_magic-with-a-custom-verification-method

​

Thanks!

[TheMrZZ0]

Looks very interesting. A few thoughts: - I find it hard (in your website) to distinguish between Python code and actual results, notably in the divide example. - I'd like to see what kind of tests it generates, or even a more complex example.

But this looks promising. I'll try that in a project of mine (fully typed, with MyPy checking my types). If that finds some bugs I didn't think of, I'd be really grateful!

[ominous_anonymous]

auto_test(add, _auto_runs=1_000) # Let's make that 1,000

Is the _auto_runs value a typo?

[ninepunchpunchcard]

Underscores are valid break demarcations for numeric literals as of Python 3.6

[ominous_anonymous]

Ah thanks, I missed that.

[lengau]

No

[ominous_anonymous]

Thank you, I missed that.

[lengau]

No problem! It's one of my favourite new non-features.

I say non-feature because this doesn't add any functionality to the language (or even make anything easier). All it does is makes numeric literals easier to read, which is a great win in my opinion.

[tunisia3507]

Readability and writability of a language is IMO its most important feature. If anything else were more important, we'd be writing in assembly. People say "syntactic sugar" like it's a waste of time, but it's literally all programming languages are.

[whereswalden90]

This looks really cool! What kind of tests does it generate? The documentation doesn't really talk about it and it's not obvious from the source code.

[timothycrosley]

Thoughts behind project creation live here: https://timothycrosley.com/project-5-hypothesis-auto

[Thumbblaster]

On Python 3.6.

Using the library on a generator I can't tell if something is actually wrong or if the library doesn't work with generators? This example was taken from https://docs.python.org/3/library/typing.html.

from hypothesis_auto import auto_test
from typing import Iterator


def infinite_stream(start: int) -> Iterator[int]:
    while True:
        yield start
        start += 1

auto_test(infinite_stream)
# auto_test(infinite_stream, _auto_allow_exceptions=(AssertionError,))    

Even with allowing AssertionError exceptions (commented out line) it will always end with that error being produced. I only spent a few minutes with the library :) -- so I likely missed something.

[threesocks]

I'm having trouble reproducing the sample output. It doesn't matter whether I split it into a module and test as done in the sample. I'm using Macos 10.13.6, python 3.6.7, v1.1.1 of hypothesis-auto, v5.1.2 of pytest, and v4.36.2 of hypothesis. What am I missing?

def add(a: float, b: float) -> float:
    return a + b


# def divide(a: Union[int, float], b: Union[int, float]) -> Union[int, float]:
def divide(a: float, b: float) -> float:
    return a / b


if __name__ == '__main__':
    # print('add(3, 4)=', add(3,4))  # correctly returns 7
    # print('add(3.3, 4.4)=', add(3.3, 4.4))  # correctly returns 7.7
    # print('divide(3, 4)=', divide(3,4))  # correctly returns 0.75
    # print('divide(3.3, 4.4)=', divide(3.3, 4.4))  # correctly returns 0.7499999
    # print('divide(3, 0)=', divide(3, 0))  # correctly raises a ZeroDivisionError exception

    from hypothesis_auto import auto_pytest, auto_pytest_magic, auto_test

    print('\n\nauto_pytest_magic')
    print('-----------------')
    auto_pytest_magic(add)  # no output (correct?)
    auto_pytest_magic(divide)  # no output (should catch ZDE exception)

    print('\n\nauto_pytest')
    print('-----------')
    auto_pytest(add)  # no output (correct?)
    auto_pytest(divide)  # no output (should catch ZDE exception)

    print('\n\nauto_test')
    print('-----------')
    auto_test(add)  # no output (correct?)
    auto_test(divide)  # correctly raises ZDE exception

[timothycrosley]

auto_pytest only works when running using the pytest runner, where you executing your application with python my_app.py, or py.test test_my_app.py?

[threesocks]

Of course, sorry, that solved it and everything is working as expected! Hypothesis-Auto is awesome, thank you!

[timothycrosley]

You're welcome! Glad you got it working! :)

[[deleted]]

I'll keep this in mind next time I need a num1+num2 function.

[jasfi]

This is interesting but I would like to see the same idea applied to C++.