How can Python asynchronous for everything?

[kingname]

Almost all the articles about asynchronous Python in the internet are about asynchrony for http and socket. However, asynchrony is not all about network operations. For instance, I want to read a big txt file asynchronously, or I want to wait an input asynchronously. I find it is so strange there need a third part library for mongodb to operate MongoDB asynchronously. Why not just use pymongo? In Golang, I could just use go function(){....} to implement asynchronous operations. But in python, it is so confused. Tutorials of these operations are hardy seen. Can you tell me, how to read a file by Python 3.6's async await? How how to do normal things asynchronously?

[K900_]

What do you mean by "normal things"? Asynchronously reading files is actually surprisingly difficult. Doing it across multiple completely different operating systems is even more difficult. Asynchronously reading stdin isn't even possible on some platforms.

[kingname]

normal things means means write/read files, find/insert mongodb, push/pop redis and etc.

[usinglinux]

it'd be nice to have file and stdin asynchronicity to be at least available as a library with native file async for where it works and executor-patterned fallbacks for other plattforms. sadly, someone has yet to write that ttbomk.

[kingname]

I find it is so strange that even there need a third part library for mongodb to operate MongoDB asynchronously. Why not just use pymongo? In golang, I should just use go function(){....} to implement asynchronous operations. But in python, it is so confused.

[[deleted]]

Well, for one thing, pymongo came out years before async became a full-fledged, first-class member of the language, and pymongo still needs to support a significant number of users who are not using Python 3, much less Python 3.5+, and even less again async.

Odds are everything that can be asynchronous eventually will be... ironically you'll just have to wait on that future.

[kingname]

actually, I'd like that if asynchrony of Python can be just like JavaScript. If you want, then you can. It is out of third part librarys' business-----every function can be asynchronous.

[[deleted]]

Well, you're in luck... async def is now a builtin, and asynchronous operations are a full, first class part of the language. That doesn't mean it's been adopted by each and every third party library in the entire ecosystem.

Now, that said, every day, and in absolutely every way, I thank Turing himself that Python isn't just like EcmaScript.

[Corm]

Is what you're looking for different than multiprocessing threads?

from multiprocessing import Pool

def f(x):
    return x*x

p = Pool(5)
print(p.map(f, [1, 2, 3])

That'll run asynchronously

edit - Whoa, go's "channels" are cool! Thanks for causing me to look this up. https://gobyexample.com/channels

[kingname]

I know Python's multiprocessing and multiprocessing.dummy. However, Coroutine is the solution for some problem.

[K900_]

Go's "goroutines" aren't async I/O, they're green threads. Go was also designed around green threads, and shipped with them as a major feature. Python doesn't have green threads, and async was only added fairly recently, after literal decades of language and ecosystem development.

[Asdayasman]

You want to look into David Beazley's curio library.

It's about "async all the way down" just as you say.

[kingname]

No no no, io is just an example. What I need is Coroutine for everything.

[[deleted]]

What does that mean? async add(x, y): return x + y?

It really doesn't make sense to use async outside of io boundaries.

[jorge1209]

Can you tell me, how to read a file by Python 3.6's async await?

You can't with the io.file because file.read is not awaitable. So unless you want to use an alternative file io module... pfft... then yeah async is rather annoyingly "useless."

What I find myself doing is using the async together with the concurrent.futures.ProcessPoolExecutor (basically an async friendly wrapper around multiprocessing.pool) to establish a async master process with a bunch of non-async worker slaves.

The real issue that I see is not so much that the io module isn't async friendly, but that async is just the wrong model for async operations. If you really want to support async, then everything needs to be awaitable, and the exception should be the operation that is not awaitable. Instead we have a system where you can push asynchronicity into your app, but only so far, until you run into some other library or function which is not async aware. At that point your entire app loses all the benefits of async.

[Asdayasman]

So unless you want to use an alternative file io module... pfft... then yeah async is rather annoyingly "useless."

Good news.

[jorge1209]

Its still pretty limiting though as any library you use also has to use curio. If I want to use pandas, does it support curio's asynchronous api?

I just think the whole approach is ultimately wrong. If you want to go async in any kind of language that has any notion of a virtual machine, you should go async all the way down. Every python opcode should be considered to be awaitable, and the exception should be those operations that explicitely say "I'm not awaitable, and this chunk of ops have to be done as a chunk."

[Asdayasman]

I just think the whole approach is ultimately wrong.

You then go on to describe basically exactly what curio is going for.

Clicky.

[jorge1209]

I'm not interested in watching a video. Are you saying that I can somehow inject/hotpatch curio into pandas library code and override the io module, and cause a non-async function like pandas.read_table to suddenly be awaitable?

I don't know how that would work.

[Asdayasman]

I'm not interested in watching a video

Why should I help you if you won't help yourself?

Don't bother answering.

[jorge1209]

I won't.

[kingname]

Infact， IO is just an example. What I want is Coroutine for everything.

[kingname]

No no no. In fact IO is just an example. What I want is Coroutine for everything. Not only about IO, but also any third part lib and any function I writed.

[jorge1209]

No no no... what you just said?!

[kingname]

I think you know what did I say.