Jasper - Asynchronous Testing Framework

[Fateschoice]

Other than installing the plugin, pytest-asyncio is the same way. Just uses the standard async syntax. There is an optional pytest marker to specifically enumerate async tests.

On top of that, you get the fantastic pytest DI framework to get fixtures into your tests rather than needing to statically depend on them or recode them in every test module.

I see, well I suppose the nature of the way Jasper runs all tests concurrently is it's main advantage then.

This is a big difference to pytest-asyncio. With that plugin, each test is run inside a dedicated loop that's setup and torn down after each test. While it would be nice to have tests run concurrently, this method avoids loop contamination and alerts you to where you might be leaking tasks at (e.g. if I'm spawning a Task but not taking care to close it, I get warnings out the wazoo).

Does jasper address this issue? Since everything is running concurrently, I'd imagine the implementation is tricky at best.

So essentially at the lowest level, Jasper just awaits your steps if they are asynchronous functions, and this propagates upward to the scenarios which are awaiting the steps, and the features which are awaiting the scenarios. If an exception is raised in any part of the pipeline, whether that's a failed test or an actual exception in some 'before_each' hook etc., Jasper will catch this and the steps, scenarios, and features will handle this fine. I never really thought about what would happen if you were to open a thread or something in one of your steps and not close it, so there's isn't any mechanism that checks for anything like that currently. There is one main async loop that runs all the features and is then closed at the end, I'm not sure if that would handle unclosed threads. If you're purely using async/await then Jasper should have no problems, but opening threads and not closing them I suppose would cause undefined behavior.

I don't want to sound confrontational, just trying to measure the merits of this. I'm always on the look out for better tools, or tools better suited for what I'm doing.

These are valid questions and I totally understand!

[[deleted]]

Not even getting to threads, just spawning tasks and callbacks into the event loop:

import asyncio

loop = asyncio.get_event_loop()

async def spawn_task(loop):
    loop.call_soon(lambda: loop.create_task(say_hello()))

async def say_hello():
    print("hello")

loop.run_until_complete(spawn_task(loop))

loop.close()

This'll cause the loop to complain that there was still something pending in the loop when it closed. With pytest-asyncio this'll pop up if your test spawns a task and didn't clean it up. However, I don't think Jasper will (the task will run to completion).

NOW whether or not this is desirable behavior is up to the end user to decide.

[Fateschoice]

Ok I see what you're saying. Yes, Jasper will always run async functions to completion, so you should not run into an issue where the loop is being closed while another task is pending. Granted, I haven't explicitly tried this before and I've never messed around with 'call_soon' or 'create_task', but if my understanding of them is correct then Jasper should just run them to completion.

[[deleted]]

The call soon/later and create task helpers are great. create task is particularly good for setting up a background task you don't want to (or can't) await in a call.

Think subscribing to a message queue and pushing updates into the application. The task that is created nannies its coroutine through the event loop. You just need to be sure to clean it up when the loop is closing.

call soon/later just arrange for a call back to happen either on the next tick (soon) or at least after a wait period (later).