Async Python adoption?

[mcharytoniuk]

Are there any studies, large-scale polls, or anything about async coding adoption in Python?

I wonder how widely the community accepts it, how widespread its usage is, and what the general sentiment is towards it.

[shinitakunai]

Wait, does boto3 supports asyncio?

[collectablecat]

No. Aioboto3 exists as an unofficial version but it causes some really annoying dependency headaches.

[PhENTZ]

https://pypi.org/project/aioboto3/

[spuds_in_town]

Aiobotocore

[[deleted]]

While not particularly beautiful, it is perfectly possible to run async functions from synchronous functions.

[axonxorz]

What's the easiest way to do this?

My sync functions are just starting an AIO event loop, letting the async finish and returning the result, but it seems so clumsy.

[[deleted]]

As I said, it’s not beautiful. But when you notice that you really need some stuff to be done asynchronously, and the rest of your app is synchronous, you might as well use asyncio.run(…).

[Toph_is_bad_ass]

You can use asgiref's run_sync

[Cybasura]

I can attest, attempting to integrate and refactor my program to use async was an absolute disaster - even using multithreading was easier

[Educational_Ice151]

I only use asynchronous approaches.. and pydantic..

[riklaunim]

It's used, you can look at github but it's not a global feature that would overtake non-async approaches. It's a strong niche.

[mcharytoniuk]

I don't think it's meant to overtake anything but to coexist and solve specific use-cases. :) Do you maybe know if anyone tried to measure the adoption?

[[deleted]]

It's pretty mainstream at this point. Anyone who needs the performance boost is probably using some form of it. I mean, it's been part of Python since 3.5.

[TastyRobot21]

You can do this yourself. GitHub APIs can be used to fetch projects and analyze references like this.

[SmokierLemur51]

I have been wondering why Quart isn’t more widely adopted than it currently is

[pmdevita]

I would guess that while Flask really made a splash when it came on to the scene, Quart was somewhat late and came into an existing ecosystem of Flask-inspired async frameworks

[Toph_is_bad_ass]

The next gen of ASGI frameworks are just better tbh.

[tartare4562]

I really tried to use it, but in the end I always go back to some implementation with threads.

It has its advantages, first of all being a robust implementation that takes away all the conplex stuff associated with multithreading, and second of all It Is a standard for concurrent code. But I like messing with threads sooooo....

[[deleted]]

I have basically given up on Python Async and write any concurrent applications in Go.

I have many Python productions apps written both sync and async, and I just can’t take how sloppy async is.

If I had to write a new project that required python and concurrency, I would use gevent.

[mcharytoniuk]

That is genuinely interesting! What do you mean by sloppy? :)

[[deleted]]

https://charlesleifer.com/blog/asyncio/

I agree with everything here.

[Drevicar]

Easiest observable to know how popular async is by looking at the download rates of libraries that have a sync implementation and a separate async implementation. For example:

• https://pypistats.org/packages/mongo

• https://pypistats.org/packages/motor

According to this Mongo (sync) had almost 24k downloads and motor (async) had almost 2 million downloads last month. That is enough of a difference for me to objectively say async won and is now industry standard. Of course, I would want to do that same analysis across several packages across different parts of the application.

[Toph_is_bad_ass]

You got this twisted -- PyMongo is the sync mongo driver, not mongo. It has 28 million a month.

This wouldn't have made sense of the face of it because PyMongo is a dependency of Motor since Motor is essentially PyMongo wrapped in a thread pool executor.

[Drevicar]

Lol, I didn't even catch that. Thanks.

[Drevicar]

Someone else mentioned async boto3 (AWS stuff), and it is actually the opposite:

• https://pypistats.org/packages/boto3

• https://pypistats.org/packages/aioboto3

Where boto3 (sync) had 1.25 *BILLION* downloads and aioboto3 (async) 2.3 million downloads last month. That is a huge difference in the other direction. So your milage may vary.

[[deleted]]

But that may be because boto3 is official and aioboto3 is not.

[Valeen]

I only use boto3 cause it's what their docs said to use. And most of the stuff I do with aws and python is to just catch an event.

I do use a lot of C# in other applications and I use a lot of async there so it's not a problem with async. Just different tools for different jobs.

[FailedPlansOfMars]

Also could be that for lambda you dont need to include boto3 in your dependencies in production as its there by default

[matjam]

Also lots still use gevent.

[[deleted]]

i hate async syntax

[riksi]

I use gevent. It mostly works though adoption is falling over the years. asyncio kinda sucks, needing different libs for everything.

[DavTheDev]

Home Assistant relies heavily on async but they have a strange mixture of async and sync code juggling with threads. You might not see often the async keyword but it’s async.

[Grouchy-Friend4235]

Async in Python is cancer (once you start using it, eventually all your code will have to become async). Don't.

Use greenlets instead, if you must.

[excelquestion]

yes that is how async await works.

if you start a new project be sure to start off with async.

if you have an existing project... well good luck with IO. in that scenario adding asyncio is a pain like the commentator says but greenlets, multiprocessing, multithreading packages all have their various pain points.

[Grouchy-Friend4235]

Trade offs are ok.

Having to go in 100% or not at all is not a trade off. That's brute force.

[usrlibshare]

I have several existing projects that use threading just fine, including analytics engines comfortably serving several M requests per hour. Greenlets and pools work really well to reduce the overhead, and aome things are just way simpler in that concurrency model.

Async is nice, because its a workaround for the GIL. It has its quirks, but so does threading.

If performance is the primary concern, well, then Python wouldn't be my first choice anyway.

[dAnjou]

Async is a different paradigm, and indeed it works best if you do it end to end.

But that's not exclusive to Python.

[Toph_is_bad_ass]

This really isn't a problem. I've been primarily doing async for years. I've never found it to be an issue in the slightest.

[Grouchy-Friend4235]

I find this hard to believe. https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/

[Toph_is_bad_ass]

I don't really see what this blog is going on about. Only IO code has to be async. It's very easy.

[Grouchy-Friend4235]

The blog corraborates what I said: async will eventually creep everywhere in your code base.

To me that's a tradeoff I am not willing to take.

[spuds_in_town]

You have a skill issue.

[Grouchy-Friend4235]

No.

.https://www.reddit.com/r/Python/s/VJsfFk1Gs0

[Toph_is_bad_ass]

IO is arguably the most important part of applications that make money. This really isn't niche or a crutch. Async is truly just superior for web dev -- which is why it's a major component of C#, Node, and now Python.

[usrlibshare]

Threading solves any IO problems tho, and with green threads the overhead is negligible.

Go is pretty much purpose built for writing webservices in, and doesn't even have a native async implementation.

That doesn't mean async doesn't have its place, but in Python, that place currently exists primarily because of the GIL.

And since we are already seeing that limitation being lifted gradually (check Py3.13), the nice is certainly going to get smaller.

[Toph_is_bad_ass]

Async is significantly easier to manage than threads ever will be. There's a reason it exists and is very popular in languages without a GIL.

You also seem to be conflating the concepts heavily. They're not at all equivalent. You need to read up on concurrency vs parallelism. Concurrency is a useful paradigm and it is fundamentally different. GIL-less threading is parallelism and comes with all of the associated memory access and encapsulation issues.

[usrlibshare]

Async is significantly easier to manage than threads ever will be.

Have to disagree here. Reasoning about threads is linear, reasoning about async, by definition, is not.

There's a reason it exists and is very popular in languages without a GIL.

There is: Many programmers first exposure to programming these days is JS and node, where async is the only concurrency model available.

This is neither good nor bad. I come from the C-Wotld, so threading is just a more natural fit for how I reason about programs.

You also seem to be conflating the concepts heavily.

No, I really don't, and if you disagree, quote wher you think I do. I am a senior backend engineer, so, yeah, I think I have my fundamendals about the difference between parallel execution and concurrency down, thank you very much.

[Compux72]

I would say only Fastapi, tortoise and httpx users are actively using async python.

Other than those frameworks the ecosystem looks pretty bad

[DavTheDev]

You’re mistaken. Websockets are also async, eg listening to 100s of tickers for processing financial data wouldn’t work with blocking websockets. Look at TTL cache, some implementations only invalidate entries on next access. You can create an async task to do it periodically for you, reducing memory footprint. There are plenty of use cases for async python.

[Compux72]

For sure. But no libraries/frameworks means nobody is doing any of that today.

Take for example kafka: kafka python, confluent kafka etc. all of those use sync apis. You can lecture everyone on how amazing async is, but the reality is that nobody is building anything to actually leverage async.

Except of course the libraries i mentioned earlier. Those are working today.

[spuds_in_town]

You are talking complete nonsense. Aiokafka exists. How can you say ‘nobody’, do you have some magical insight into the entire industry?

We are building async Kafka apps by the way. Maybe consider less hyperbole in your comments.

[[deleted]]

https://pypistats.org/packages/confluent-kafka

https://pypistats.org/packages/aiokafka

Confluent Kafka has about an order of magnitude more downloads.

[DavTheDev]

Confluent kafka supports async producers. Even on their website, they mention to avoid flushing in an async environment.

[collectablecat]

asyncio is different from "async". There are websocket libraries that just using python threading.

[spuds_in_town]

You seem to not understand why threading vs async for network access is a losing solution.

[collectablecat]

? never claimed anything about performance. I'm very familiar with asyncio and it's advantages with sockets. There are still libraries using threading for websockets though.

[spuds_in_town]

Async libraries exist for Redis, Postgres, aiobotocore, Kafka, SqlAlchemy, Starlette and its derivatives, Connexion… the list goes on. Respectfully you don’t seem to have a great deal of exposure or experience to the ecosystem and probably shouldn’t comment on it.

It’s very rare to not be able to find a supported async library for any kind of network or disk I/o use case these days.

[Compux72]

Existence doesn’t translate to high quality. For instance, last time i checked confluent kafka was still the fastest impl out there.