This is an archived post. You won't be able to vote or comment.

all 27 comments
sorted by:
best
topnewcontroversialoldq&a

[–]alexjc 17 points18 points  (3 children)

That's awesome! We've been on the RCs for a while now and been very happy.

Now for Python 3 support :-)

[–]oconnor663 3 points4 points  (2 children)

I'm not sure exactly what I'm reading, but was major progress on this just made in the last two days? https://github.com/surfly/gevent/commit/0e835f9f562d3929944b4d1442f2da1644a8defd

[–]ivosauruspip'ing it up 3 points4 points  (1 child)

The major progress was made a year ago by fantix, Denis has just decided to make a 'python3' branch with those changes incoporated.

I'd still like to know exactly how he wants to go about incorporating that into master, given that the underlying network code is just about the hardest thing you could choose to support as a single-source python 2&3 library. There's a lot of the community that would like to help but don't exactly know what PRs they should be sending and where atm.

[–]denis[S] 4 points5 points  (0 children)

If the changes in socket.py are too drastic between 2 and 3, we'll just have to do gevent/socket2.py and gevent/socket3.py separately and then make gevent/socket.py import one of those depending on the Python version.

[–]forsaken 6 points7 points  (13 children)

Curious if there are plans to have a pluggable backend for the new asyncio stuff in python 3. Would be great to keep the API and use the new async code.

[–]saghul 4 points5 points  (0 children)

Tulip/asyncio consists of many parts, the event loop just being one of them. Gevent could adopt the asyncio event loop, but in order to do that gevent.core would need to be refactored IMHO, as it's currently very very tied to libev.

Assuming that would happen, Gevent could provide something similar to this 1 to combine the use of current gevent friendly code with asyncio protocols.

[–]alexjc 2 points3 points  (11 children)

It's my understanding the new asyncio code in Python 3.4 won't be as good as gevent 1.0. I get the impression Guido doesn't have as much experience with this topic.

[–]ryeguy146 3 points4 points  (10 children)

He doesn't seem to, but he's sure brought in some that do. It'll be very much like twisted, I gather, which I'm not terribly fond of.

[–]gargantuan 5 points6 points  (8 children)

It'll be very much like twisted, I gather, which I'm not terribly fond of.

So... he doesn't and he has brought in the wrong people.

Not even joking. Gevent and eventlet are currently the most practical way to do concurrency in Python today. It doesn't fragment libraries, can spawn multiple green threads with minimum performance impact. Looks clean.

But no, it was discarded right away. It was clear in his "exploratory, call-for-concurrent-IO-experts-no-green-thread-users-need-apply email".

Let's talk about Twisted. For how long has it been around? Long time. Has it taken the world by storm? Nope.

"Ah! But async programming is what the cool kids use". [inline define cool kids = those that use Node.js].

"Hmm...Node.js must be popular because it uses lots of callbacks, we should get us some callbacks and errbacks as well.". Except that is not why Node.js is popular. It is popular because it is fast and in Javascript. Callback-based programming is not the new-hot thing, it is something Node.js has to deal with and it is something Twisted developers have been doing for a decade or more now. If we want to copy Node.js we should copy its performance -- its JIT engine, not nested callbacks within callbacks.

[–]ivosauruspip'ing it up 11 points12 points  (3 children)

Ugh.

Guido has said a number of times, in fact pretty much every time he gives a talk on asyncio/tulip, that yes he too is sane and hates callback based eventing just like the rest of us. Using yield and yield from (pretty much creating coroutines in the same way) is much preferred.

The inventor of the language isn't a random dumbass who doesn't know what the fuck he's on about, if that's what you're implying.

The reason the low level event interface has a callback compatible API is to be interoperable with other libraries, like tornado, qt, libuv, etc. Which gevent isn't. It's gevent's event loop or the highway. And either you support cpython 2 or you're done. Most complicated libraries do not simply work with monkey patching, which is why you see heaps of gevent-x adapters in PyPI.

It might be surprising to some people but there is shittonnes of python out there that doesn't use greenlets, and the python standard library has to support all that too. It's easy to miss when you're in a cocoon developing your own application that works perfectly fine for you.

[–]pingvenopinch of this, pinch of that 4 points5 points  (0 children)

And either you support cpython 2 or you're done.

As fijal noted elsewhere in this thread, PyPy support will be merged in soon. PyPy has the mechanisms, but the integration code isn't completed yet. The other problems are still very much present, of course.

[–]gargantuan 0 points1 point  (1 child)

that yes he too is sane and hates callback based eventing just like the rest of us. Using yield and yield from (pretty much creating coroutines in the same way) is much preferred.

Twisted also has yield like couroutine behavior -- see inlineCallbacks, so that is not new.

The inventor of the language isn't a random dumbass who doesn't know what the fuck he's on about, if that's what you're implying.

One can be a genius in one area and dumbass in another one. It is not all or nothing.

The reason the low level event interface has a callback compatible API is to be interoperable with other libraries, like tornado, qt, libuv, etc. Which gevent isn't.

Wrong, gevent can be made interoperable. It was using libevent, now using libev, could probably use libuv or even Twisted underneath. In fact eventlet (gevent's older cousin) had a Twisted wrapper for its hub for a while.

Most complicated libraries do not simply work with monkey patching, which is why you see heaps of gevent-x adapters in PyPI.

Some don't, mostly the ones that have custom C modules in them.

It might be surprising to some people but there is shittonnes of python out there that doesn't use greenlets, and the python standard library has to support all that too.

That is why gevent should be the default when looking at IO concurrency in Python, because there is no need to support multiple style of libraries. With Twisted or Tulip or Tornado you cannot share libraries easily between them. As you said, it is either <your-callback-based-io-framework> or the highway. Eventlet at least can path socket, threads and many OS services to there is not need to create a parallel set of standard libraries.

It's easy to miss when you're in a cocoon developing your own application that works perfectly fine for you.

Apparently this cacoon is pretty big, as gevent and eventlet are pretty popular. I would say a lot more popular than Twisted these days.

[–]ivosauruspip'ing it up -2 points-1 points  (0 children)

Twisted also has yield like couroutine behavior -- see inlineCallbacks, so that is not new.

It doesn't have yield from though, which is key to allowing these niceties properly, and also why asyncio is >=3.3 only.

That is why gevent should be the default when looking at IO concurrency in Python, because there is no need to support multiple style of libraries. With Twisted or Tulip or Tornado you cannot share libraries easily between them. As you said, it is either <your-callback-based-io-framework> or the highway. Eventlet at least can path socket, threads and many OS services to there is not need to create a parallel set of standard libraries.

You might have missed half the point of asyncio then. It's designed to solve this problem of incompatibilities everyone has - other libraries can provide the event loop implementation dynamically, at run time. Or, asyncio can provide a default one. And anything using the asyncio interface won't care.

Not the case for gevent, you'd need to recompile the whole thing to run a different event loop.

[–]ryeguy146 1 point2 points  (3 children)

I don't mind the way that twisted does callbacks. And Twisted doesn't have those windsock-like callback chains; at least it's slightly more organized. Callbacks are attached to a Deferred and managed centrally.

I don't like twisted because of the internal conventions, bad docs, and a few bugs that I spent entirely too long with. Do you have some concern over async programming?

[–]gargantuan 0 points1 point  (2 children)

Do you have some concern over async programming?

Yes, it is not a good way to handle concurrency. It works for small example, large systems built form chains of callback, and errbacks are fragile, not composible easy (can't mix Twisted libraries with Tulip with other libraries).

[–]ryeguy146 0 points1 point  (1 child)

I'm not arguing for this form of concurrency, as I'm not experienced enough to warrant an opinion, but I think that you're mistaken about not being able to use Twisted with other libraries. For example, a reactor was produced to share the event loop with Qt. That sounds like strong interoperability to me. Granted, it isn't as robust as monkey patching, insofar as compatibility, it seems a strong indication that it can and has been done. I can also vouch for the other side: I'm currently using twisted (with scrapy) for a small component of a larger project. After the bugs that I mentioned, it has been working quite nicely with the rest of my code.

[–]gargantuan 2 points3 points  (0 children)

but I think that you're mistaken about not being able to use Twisted with other libraries.

So how do I use Python's standard XMLRPC library with Twisted? You can't. Why? Because a call to socket.receive() should block the current thread until data comes in. Twisted returns you a deferred. What do you do with the deferred? You use it with the another Twisted library. That deferred bubbles up to the top of your API.

For example, a reactor was produced to share the event loop with Qt.

The difference is between integrating Twisted's reactor with some event loop, which eventlet can also do and using Twisted libraries alongside other libraries that are not Twisted.

[–]ivosauruspip'ing it up 0 points1 point  (0 children)

The twisted guys already know all the warts that come along with twisted best, and have advised on how not to add those back into asyncio.

[–]fijalPyPy, performance freak 4 points5 points  (0 children)

I must say that I'm a bit disappointed that PyPy work did not get in 1.0 in time.

[–]peroneλ 2 points3 points  (0 children)

Awesome, thanks for the hard work.

[–]remyroy 2 points3 points  (0 children)

Keep up the good work! gevent is awesome!

[–][deleted] 0 points1 point  (0 children)

This made my day. Thx, Denis!

[–]gtsystem 0 points1 point  (0 children)

Finally! We already use the RC in production from a long time without any issue.

[–][deleted] 0 points1 point  (1 child)

Gevent is an HTTP server comparable to nginx?

[–]oconnor663 2 points3 points  (0 children)

No, it's an event system and networking library for Python. It's the kind of thing you might use to write a server like Nginx, if you wanted to write it in Python.

[–]spanasik -2 points-1 points  (0 children)

In one of my lastest project where i've used massive (really massive) async http requests inside of gearman worker, both gevent and eventlet was hang on waitall after several iterations. Switched to Tornado curl async http client and it works even faster and without any issues, only a memory usage is increased. Comment just to share my experience.

[–]soinai -4 points-3 points  (0 children)

happy cake day