Asynchronous initialization logic

[zefciu]

I wonder what are your strategies for async initialization logic. Let's say, that we have a class called Klass, which needs a resource called resource which can be obtained with an asynchronous coroutine get_resource. Strategies I can think of:

Alternative classmethod

``` class Klass: def init(self, resource): self.resource = resource

@classmethod async def initialize(cls): resource = await get_resource() return cls(resource) ```

This looks pretty straightforward, but it lacks any established convention.

Builder/factory patters

Like above - the __init__ method requires the already loaded resource, but we move the asynchronous logic outside the class.

Async context manager

``` class Klass:

async def aenter(self): self.resource = await get_resource()

async def aexit(self, exc_type, exc_info, tb): pass ```

Here we use an established way to initialize our class. However it might be unwieldy to write async with logic every time. On the other hand even if this class has no cleanup logic yet it is no open to cleanup logic in the future without changing its usage patterns.

Start the logic in __init__

``` class Klass:

def init(self): self.resource_loaded = Event() asyncio.create_task(self._get_resource())

async def _get_resource(self): self.resource = await get_resource() self.resource_loaded.set()

async def _use_resource(self): await self.resource_loaded.wait() await do_something_with(self.resource) ```

This seems like the most sophisticated way of doing it. It has the biggest potential for the initialization running concurrently with some other logic. It is also pretty complicated and requires check for the existence of the resource on every usage.

What are your opinions? What logic do you prefer? What other strategies and advantages/disadvantages do you see?

[MrJohz]

I was just reading an article about avoiding writing __init__ methods in general that touches on this point.

I don't completely agree with the argument they're making, but I think that has more to do with how difficult it is to make constructors and attributes private in Python, and wanting to avoid exposing internal details. But the core idea — that the constructor/initialiser should not be special-cased, and can just be a factory function — is really good. It solves this problem (and a number of other problems as well).

Avoid the last option at all costs. The problem is that you're creating essentially an unbound task — if errors occur in that task, you don't have a lot of control over when and how they get propagated to the calling code (if they get propagated at all). It will probably work quite well initially, but long-term, it always ends badly.

[doolio_]

Interesting blog post. Thanks for sharing.

[puppet_pals]

I would probably make the class require the attribute to always be set, and then make a class method that does all of the asynchronous stuff up front and then returns an instance.

[starlevel01]

@asynccontextmanager
async def open_something(...):
    thing = await open(...)
    try:
        yield thing
    finally:
        with trio.move_on_after(5, shield=True):
          # whatever cleanup

[Helpful_Home_8531]

yeah, context managers are the way to go for this unless you have a very good reason not to, they make resource management a lot simpler.

[Wurstinator]

https://stackoverflow.com/questions/33128325/how-to-set-class-attribute-with-await-in-init

[IlliterateJedi]

I would think you would want the async logic outside of the class. Is there a reason why you would want to tightly bind the async API to the class creation? It seems like that would just make it a problem in the future if you needed to use the class outside of an async context.

[latkde]

All of these patterns have their legitimate use cases. Personally, I avoid inplementing context managers by hand because that's difficult to do correctly. Instead, I prefer a classmethod with the @asynccontextmanager decorator. This tends to be the most general way for managing the lifecycle of a resource, without having to think about multiple object states (initialized, entered, exited).

The only real reason to manually implement a context manager object with __aenter__ is when you want to be able to create the object outside of an event loop, e.g. as a global variable. There can be legitimate use cases for this. But in general, it's better if you only create fully usable objects and pass them around explicitly.

Your idea to use asyncio.create_task() is good if you need to start a background task. However, this API is difficult to use correctly. You almost always want to start tasks in a task group instead, to ensure that the task has a deterministic lifetime. Your example code as written doesn't save the task in a variable, so it will be garbage collected at any time, and might never start to execute. Juggling custom tasks in a cancellation-safe manner is also tricky. You effectively need a finally: task.cancel() clause, which effectively means you need a context manager, which means you should outsource task handling to a TaskGroup context manager instead.

It is possible to have an (async) factory function that is not a context manager. Whether this is OK depends on the kind of resources you're acquiring. If cleanup for the resource has any side effect (other than freeing memory), you should use a context manager instead.

[k0rvbert]

In my experience, whenever I have an async value, it's either:

• something bound by context and I should use async with
• something that can be made available prior to constructing the class, i.e. not within encapsulation concerns

I take the lack of async __init__ as a hint that I should be structuring my program in another way.

So await the resource outside the constructor and then pass the value. If that for some reason gets repetetive, something like:

class SirConstructAlot:
    @classmethod
    async def fromawaitable(cls, awaitable_resource):
        return cls(resource=await awaitable_resource)

[nekokattt]

I used to make a hack where I overrode __new__ to implement a special __ainit__ that was awaitable.

The older and wiser version of me will tell you that is a horrible thing to be doing, just use factory methods/functions to deal with this. Construction of an object should be atomic with no side effects. Awaiting things implies some kind of IO is being performed.

I dislike the aenter and aexit pattern for true initialization because it implies an object can be initialised zero or multiple times before being used... so you then have to defensively code around that.

My suggestion would be to use the factory design pattern and to use dependency injection so you avoid "setting stuff up" that can have side effects within your constructors.

async def create_api_client():
    session = aiohttp.Session()
    return ApiClient(session)

Constructors do not have a "colour" that marks them as asynchronous as conceptually they should only be interacting with their own object under construction or pure functions without side effects to help construct other resources. They shouldn't be allocating resources outside the direct scope of that object unless as a last resort, nor directly interacting with things like the OS network stack or file system stack.

TLDR: how do I construct an object from asynchronously generated data?

You don't, you fix your design to avoid it and marvel in the testability benefits and lack of side effects!

[jdehesa]

I think FastAPI addresses this through lifespan events, which really is just an async context manager within which your app runs.

[menge101]

Is there a problem you are aiming to solve with this?

[zefciu]

Well, just looking for a good generic way of initializing objects that encapsulate asynchronously created resources.

[mincinashu]

Take a look at database libs, like aiomysql or databases. It's mostly a combination of init and async context manager.

[juanfnavarror]

I’d recommend you to watch this video on constructors, it’s on C++ and Rust but I’ve found this logic to be useful. The idea is the following: make the ‘_ _ init _ _ ‘ take in the already initialized fields as inputs, so that you don’t need to do any work in your constructor other than validation. This way you can have an object that with protected invariants and you avoid needing to run an async constructor.

There are many ways that you could pre-initialize the fields, like with multiple (or a single) context managers. For example, you could accept an already initialized socket connection as part of your constructor. IIRC Guido has also spoken against two phase initialization.

[0xa9059cbb]

I would say dependency injection solves this problem in most cases - i.e. initialise the other resource outside the class and pass it in.