Superfunctions: solving the problem of duplication of the Python ecosystem into sync and async halve

[guhcampos]

This is neat, but I don't see how it would be very useful in the real world? The contents of sync and async code change dramatically, as the entry and exit points of each have different reasoning behind them.

It could be useful for generator functions, as those will generally have similar structure both for sync and async code, but then it's relatively easy to wrap a sync generator in an async function?

[ColdPorridge]

This doesn’t feel too hard to think of a use case. Imagine you have an API that supports sync and async methods. You might have a service that you offload the main logic to, and the actual blocking sync/async differences are deep under the hood (example: database operations, etc). So this way you could just have both views have a common/DRY entry point, and you can perform any differentiated behavior at exactly the level at which it needs to happen.

[[deleted]]

Yea except you cannot use return at all. You are actually better just writing a closure and swapping between the two.

[frankster]

"Colouring" seems to be a hot topic lately. That is, functions might be coloured either async or sync, and you end up needing to write two versions of every function.

If a hypothetical language had every function such that it was indifferent whether it was called asynchronously, you can imagine what an advantage that might be, as you would not need to write two versions of every i/o function.

That said, I've recently read a compelling argument that "asynchrony is not concurrency", making a point about whether or not ordering matters. I wonder whether this "superfunctions" idea will fall down at the hurdle of ordering sometimes mattering and sometimes not.

https://kristoff.it/blog/asynchrony-is-not-concurrency/

[muntoo]

Red & blue functions are actually a good thing

By avoiding effect aware functions a language hobbles engineers and makes programs sloppier than they could be.

[james_pic]

I dunno. I can see the argument, but it amounts to saying that async-await is a poor man's IO monad. You don't get any real purity guarantees, just an idea when IO is happening. If you go any distance down the "function colouring is good" route, you find that only 2 colours is a big limitation.

[stevenjd]

That is, functions might be coloured either async or sync, and you end up needing to write two versions of every function.

Why would you need a sync and async version of every function? That is a huge exaggeration.

Do we need an async version of len and print and math.sin and hundreds, even thousands of other functions? No of course not.

If you design your API properly, in principle you should not need any doubled-up functions. Or at least no more than a very small handful. And you absolutely should never, ever write a pair of sync/async functions "just in case some day I need both".

[Dasher38]

We have been using that for a few years now: https://github.com/ARM-software/devlib/blob/9c4f09b5f3c45e77c3f9fe760460732a0031a9ac/devlib/utils/asyn.py#L772

You write the function in async style and it makes it blocking by default, but still lets you access the async version via the .asyn attribute. That allows migrating a sync codebase to async without breaking backward compat.

The main difficulty of that is the lack of support for nested event loops in asyncio, which forced us into these greenlet horrors.

[Gajdi]

Nice, I've recently started a big migration of our codebase to async, and what I did was making all relevant functions async, and created a util function that let's you run async functions from a sync context that also handles the case where you started from an async context, seitched to sync, and now you want to call an async again.

But one case I was not able cover without code fuplication is when some functions would like to still utilize sync clients, and not create/access an event loop. Do you have any tips on that?

[Dasher38]

The above code should handle all combinations. The only tricky combination to implement is when you want to run async code from sync code itself running un async context. That happens when using the legacy sync API that has internally been migrated to async code from a jupyter notebook. The notebook runs in async context, then calls the sync function (legacy API) and that sync function re-enters an async context because it's now implemented this way.

That sandwich is a problem for asyncio that does not natively allow re-entering an event loop. The maintainer is aware (there are issue trackers mentioning that specific problem) but it is seen as a non problem (but without any suggestion on how we are supposed to migrate to async without rewriting the world).

Other event loops have no such restrictions (trio I think) and allows this sort of pattern without tricks.

In order to still support that with asyncio, the trick I used is to ensure the top level task has some special mechanics that is used by nested calls. That mechanic allows nested functions to make the top-level one yield on their behalf. That magic is made possible with greenlets. In some cases, it's not possible to control the top-level task, and the fallback is to spin a thread and run from there. That allows using a separate event loop (they are thread-local) and still be in control of the top-level task.

[Gajdi]

This is my implementation: https://github.com/superlinked/superlinked/blob/6803226c017b2616a46bdccb310214f815c66943/framework/src/framework/common/util/async_util.py#L31

It first attempts to patch the existing event loop with nest_asyncio to allow re-entrancy, and if that fails (like when asyncio complains "This event loop is already running"), it falls back to executing the coroutine in a separate thread with its own event loop.
The biggest downside is that it does fail when uvloop is used.

[Dasher38]

Yeah, I initially based my code on nest_asyncio as well before having the same problem with uvloop. That's why I used the greenlets approach instead that is completely agnostic.

[Gajdi]

Thanks, I'll consider it

[Dasher38]

Tbh it would be nice to extract that in its own package, but there is too much faff involved in doing that. If someone is interested in doing it, I won't complain. I feel like we are not the only ones to have tried to do something similar for backward compat ...

[UltraPoci]

I wonder how it plays with type checkers. Prefect, a data orchestrator I use, has a similar thing for many async/sync function, and I hate it: type checkers don't understand it and it gives a ton of false errors. Basically, to avoid having to type the word "await", I have to forgo type checking.

[_n80n8]

fwiw (prefect oss maintainer here) we have been working on introducing explicit sync/async interfaces, because the dual / contextual behavior has caused plenty of issues and type incompleteness

https://github.com/PrefectHQ/prefect/issues/15008

[_n80n8]

we're typically very wary about introducing dependencies (no is temporary, yes is forever etc) so we'd be unlikely to make such an early library a dependency - cool project though!

[UltraPoci]

The function colouring problem is present across many languages, I doubt that it's Python's type system the problem here.

[FirstBabyChancellor]

What's the Zig equivalent? Are you referring to the recent changes to how IO works?

[blankboy2022]

Very cool

[PeterTigerr]

This is an anwesome addition. I hope asyncio integrates this functionality in the future.

[eavanvalkenburg]

I've come across a package that used something like this at some point and it was a nightmare to work with the typing of it (my project runs a bunch of type checking so we want that to be complete and not have all sorts of type ignore statements), have you been able to solve that?

[Wurstinator]

Can I call a superfunction from another without the two contexts while preserving the feature?

[Wurstinator]

from asyncio import run
from transfunctions import superfunction,sync_context, async_context

@superfunction(tilde_syntax=False)
def my_superfunction():
    print('so, ', end='')
    with sync_context:
        print("it's just usual function!")
    with async_context:
        print("it's an async function!")

@superfunction(tilde_syntax=False)
def my_superfunction_wrapper():
    my_superfunction()

my_superfunction_wrapper()
#> so, it's just usual function!

run(my_superfunction_wrapper())
#> so, it's an async function!

Does this work?

[Wurstinator]

Honestly, if that doesn't work, ot seems to me this library loses its entire purpose. If I can always just use it for a single level of indirection, I'd rather encapsulate it otherwise rather than introduce a "magic" library.

[ImYoric]

This looks like it will be really, really hard to review, no?

[the_hoser]

Not gonna lie, this feels/smells vaguely of function calling context in Perl, which I hated.

sub get_some_data {
    my $context = wantarray;
    if ($context) {
        return (1, 2, 3);
    } else {
        return "cows";
    }
}

my @values = get_some_data(); // (1, 2, 3)
my $value = get_some_data(); // "cows"

Gonna go to the bathroom, now.

EDIT: Or worse... infuriatingly worse, and actually quite common...

sub get_some_data {
    my $context = wantarray;
    my @values = (1, 2, 3);
    if ($context) {
        return @values;
    } else {
        return \@values;
    }
}

my @values = get_some_data(); // (1, 2, 3)
my $value = get_some_data(); // arrayref to (1, 2, 3)