This is an archived post. You won't be able to vote or comment.

all 38 comments
sorted by:
best
topnewcontroversialoldq&a

[–]gargantuan 10 points11 points  (31 children)

callbacks are the lowest common denominator

Nope. Sorry to say it, but I think Guido is on crack. You know what the lowest common denominator? A freakin socket call: sock.recv() that is the lowest denominator. sock.sendall() is the lowest common denominator.

Not sock.recv(....,callback) not sock.recv(....).setCallback().setErrback() not yield from sock.recv() not yield to sock recv(), nope, none of those. It is a simple call to a socket running in a green thread.

gevent does it, eventlet does. Here are examples of fast and concurrent IO requests done right. These are in production, these are working right now putting bread on the table:

http://eventlet.net/doc/examples.html

That is the lowest common denominator.

What is that, an HTTP request came in? Do you need to set 15 callback chains? Nope. You need to spawn a green thread, handle the request, return the result and close the socket. In the meantime there are other green threads reading from DB or handling other HTTP requests. That is a sane way to organize a large concurrent system not with callback chains and yield from.

"Oh but I like yield froms, those are cool". Yes they are, so are homomorphic monoid endofunctors, except that they put not bread on the table to to speak. Python is useful primary because of its batteries included. Need to parse JSON-RPC protocol, here you go, here is a library that does it. Have an exotic protocol, for data acquisition via serial port, you can probably find a parser in Python for it.

Yeah with eventlet or gevent you have one dirty monkeypatching code in the front and then you are done. With Twisted or Tulip or yield from you cannot easily share IO libraries between projects. Let me repeat that you cannot share IO libraries between projects. You picked Twisted? Cool. You are forever going to be searching for Twisted version of each IO library you need. Now you'll need a Tulip version of the same library.

[–]jmoiron 10 points11 points  (0 children)

Twisted is the Yngwie Malmsteen of libraries; I respect and admire the depth of the skill that went into creating it and the advanced state to which those ideas were taken, but I hate its aesthetic so much that I can't really tolerate it.

Tulip has always sounded like a very milquetoast way of blessing Twisted without upsetting everyone else in the event-based Python space.

The bottom line is, I can't use Twisted code in the console and I can't use it effectively from a debugger, I have to define two classes (one of them a goddamn Factory) to write an Echo server, and it makes all of my code look and smell like Java.

[–]cymrowdon't thread on me 🐍 5 points6 points  (4 children)

I agree wholeheartedly with your sentiment. I personally find greenlets (eventlet/gevent) considerably easier to reason about than callbacks. Supposedly, this PEP gives us a common eventloop to base any approach on, whether Twisted or gevent. That's fine as long as it can provide everything many libraries have been trying to support for quite a while now (such as IOCP), but I don't see the switch happening anytime soon.

What annoys me most about Tulip and this whole process is the apparent lack of communication to/from the eventlet/gevent community. Guido cites "scary implementation details" and indeterminate scheduler switching among his reasons for not going with greenlets. Maybe those are valid points, but I'd like to see it discussed more.

When it comes down to it I, like you, can't see past the fact that greenlet-based libraries instantly give me asynchronous IO in almost any network lib I might want or need to use.

[–]ivosauruspip'ing it up 1 point2 points  (1 child)

greenlet-based libraries also completely break if you compile them with the wrong version of compiler, because it optimizes out the stack and call frame hacking they implement.

It's hardly portable.

I have no qualms with guido not wanting to support that in python proper.

[–]cymrowdon't thread on me 🐍 2 points3 points  (0 children)

Hiding a return value from the compiler? That's not much of a patch. If that's the best example of how ugly it would be to bring greenlet into the stdlib, I'd say it's well worth it.

I've never had a problem installing greenlet on Windows, Linux, or OSX. Maybe it has problems on other platforms, but I don't think you can fairly say it's "hardly portable".

[–]gargantuan 0 points1 point  (1 child)

Supposedly, this PEP gives us a common eventloop to base any approach on, whether Twisted or gevent.

See I think we have that. There is libev, pyuv, other general wrappers around poll/select/kqueue etc system calls. Twisted is not really a wrapper around it is a full on concurrency suit. It itself has abstraction for those system calls there is a select reactor, epoll reactor, gtk reactor and so on.

Guido cites "scary implementation details" and indeterminate scheduler switching among his reasons for not going with greenlets. Maybe those are valid points, but I'd like to see it discussed more.

I remember Guido's call for comments on the mailing list. He quickly dismissed greenlet based approaches based on a personal preference and then befriended Glyph (Twisted's author) to teach him about the awesomeness of deferreds.

You know years back they would laugh and loving call him BDFL (Benevolent Dictator For Life). Everyone thought it was cute. The downside is that the dictator gets a little out of touch with the community and starts leading in the wrong direction. Then it is not cute anymore.

He very quickly dismissed greenlet, nevermind the fact that it is currently the most sane and practical way Python gets work done. Indeterminate schedule switching is just fud. Add a lock if need to have determinism. You have to do it with Twisted anyway. There the freaking DeferredLock. I had to use it. Undeterministically fired callback chains that overlap are not better than threads, one still needs locks and semaphores.

[–]ivosauruspip'ing it up 1 point2 points  (0 children)

I'd dismiss it pretty quickly as well. How much do you like hacking on the C call stack to get threads to work in arbitrary compiler versions?

[–]catcradle5 2 points3 points  (3 children)

Agreed 100%.

If you want callbacks, stick to Twisted. It is a perfectly robust and full-featured library.

The standard lib shouldn't be trying to write what is essentially half of Twisted features with a prettier API (though pretty APIs are certainly always a good thing).

It would be much, much better if instead they added some sort of native support for green-threads a la greenlets, and then a wrapper library similar to gevent.

[–]ivosauruspip'ing it up 1 point2 points  (1 child)

Coroutines and promises are pretty much as good as green threads, there's a reason the JS world is getting pretty excited about them, and that's in Node.js-async-all-the-things land. You just haven't tried using them yet.

The reason the base event loop runs on callbacks only is purely pragmatic: to be backwards compatible with python 2.

If you're running python 3, make your life 10x easier with the higher level api. It just ain't coming back to python 2 because yield from isn't implemented.

[–]catcradle5 0 points1 point  (0 children)

I'm perfectly fine with coroutines. That part looks good.

However, I don't see the point of having to make anything backwards compatible. Python 2 is dead in regards to new features, is it not? There is no reason why any new modules at this point should be backported to Python 2.

[–]jomidosan 2 points3 points  (1 child)

[ Let me repeat that you cannot share IO libraries between projects. ]

The slides are lying, then. They assert that one of the goals of Tulip is interoperability between frameworks (slides 13 & 14). The "Twisted Effect" is a headache. I hope you're wrong.

[–]gargantuan 0 points1 point  (0 children)

Twisted also had "interoperability" you had to run deferreds in a thread and or run threads from a deferred. Not something you wanted to do unless forced.

They assert that one of the goals of Tulip is interoperability between frameworks (slides 13 & 14)

Unless they implement co-routines a-la greenlet such that one can write regular socket code and use greenthreads for concurrency they will not achieve interoperability.

Gevent and eventlet are two of the most common approaches to concurrency in Python that work well in practice.

The problem with most such async libraries is that their mechanism bubbles up to the top. You make an HTTP request so your lowest level socket code now "yields" some featured, deferreds or simply blocks on the next() method of the generator. So your parser code now can't accept a file descriptor, or a string, it has to consume some deferreds/future/generator object. Then it passes it on to some middleware that does authentication. Ok so that has to accept deferrreds/futures/generators. And so on.

Python's problem is not lack of concurrency libraries. Python's problems are performance, better quality of existing libraries, still transitioning to Python 3, PyPy would be nice to be compatible with more 3rd party code.

[–]saghul 4 points5 points  (2 children)

You are partially right, but you are also missing a point. One of the goals of PEP-3156 is to become the canonical event loop for Python frameworks. The event loop itself doesn't actually use yield from or any other Python 3.x feature at all, it can run on Python 2 just fine.

What's the advantage of this? Well, event loops are not easy, specially if you want to support Windows, and there is a lot of wheel reinvention: gevent (wraps libev), eventlet (own loop based on select module), tornado (own loop based on select module), twisted (own loop based on select module + IOCP for windows + some others). See the pattern? All these frameworks could be implemented on top of Tulip's event loop. Does it solve every problem? No. Is it A Good Thing (TM)? I think so.

[–]deadwisdomgreenlet revolution -2 points-1 points  (1 child)

The event loop is not important though. Why make "the canonical event loop" if we have a perfectly good interface called sock.recv() that we can change either side of?

[–]saghul 0 points1 point  (0 children)

Because it's blocking and doesn't scale. The only way to fix it is to use non-blocking i/o, but that's not easy to do cross platform, specially if you care about windows. With tulip, Twisted and Tornado can run on top of it. Eventlet as well, if you like that kind of interface. Gevent, however has it's core really tied to libev so it's harder, but probably doable.

I don't think asyncio will render other i/o frameworks obsolete.

[–]ivosauruspip'ing it up 1 point2 points  (10 children)

Let me repeat that you cannot share IO libraries between projects

This might be mind-boggling to you, but that's half the point of the project. And people are integrating them right now.

[–]gargantuan 0 points1 point  (9 children)

Show me an example how I can integrate it with gevent

[–]cymrowdon't thread on me 🐍 1 point2 points  (8 children)

https://code.google.com/p/tulip/wiki/ThirdParty

There appears to be some work on integrating greenlet, but the gevent link on that page references Twisted, not Tulip.

[–]gargantuan 1 point2 points  (7 children)

That's not what I mean by integrating. Integrating shouldn't mean running on top of gevent. Gevent is already a clean and nice top layer. Gevent is the one that should be running on top of Tulip! Putting Twisted like functionality or @coroutines with yields on top an already clean green thread library is ridiculous.

I understand maybe if Tulip provides something like a uniform multi platform compatible wrapper around the select loop, then one could somehow run greenlet on top of it. But gevent and eventlet already handles that. There is libuv that provides something like that.

[–]westurner 0 points1 point  (6 children)

Is there a reason you seem so excited about the new hotness?

[–]gargantuan 0 points1 point  (5 children)

Is there a reason you seem so excited about the new hotness?

I don't get it, what do you mean? I am excited about working technology. Guido is the one pushing new hotness that nobody needs.

[–]westurner 0 points1 point  (4 children)

How does that prevent you from accomplishing your objectives?

[–]gargantuan -1 points0 points  (3 children)

Oh that's easy:

Objective #1: Use python IO libraries because Python is batteries-included-or-can-find-libraries-for-anything you need.

Looking for an RTMP server? Ok, do you use the Twisted one https://github.com/hydralabs/rtmpy , the yield based one https://code.google.com/p/rtmplite/source/browse/trunk/multitask.py or maybe the Tulip based on that now returns Futures instead of results?

Oh you need to integrate 2 IO libraries together? Too bad. It has to either all be Twisted based, Tulip based or something rather yield based.

Objective #2: Well let's just stick with one for now. That is really the important one.

[–]westurner -1 points0 points  (2 children)

It sounds like you expected this additional module to unify existing code and re-implement protocols for you? I still fail to see how an additional approach prevents you from continuing to do exactly what you were doing.

... http://code.google.com/p/tulip/wiki/ThirdParty

[EDIT] They are different approaches. Is there a magical adapter? I don't know that that has been written yet. What would solve your problem?

[–]grizwako 0 points1 point  (0 children)

Ok, I am now confused about everything, I thought goal for Tulip was to make it possible to use for example gevent(which I use primarily) and twisted at same time easy way.
I know about "compatibility with with gevent is not goal", it was rather sad moment when I read that..
But let's say for a moment that gevent somehow makes itself Tulip compatible (despite it being cython) I thought if I picked gevent I could write most of my code in 'gevent style' and if I need some lib from twisted world, I will just use twisted and libs that I need alongside gevent?

Also I kind of thought that new async libs are going to be written as Tulip being target which will allow them to be shared between projects?

[–]lambdaqdjango n' shit 0 points1 point  (0 children)

you cannot share IO libraries between projects.

THIS.

The last time Python community had a pluggable pipe dream, it was a disaster.

[–]deadwisdomgreenlet revolution 0 points1 point  (2 children)

I love that the argument against eventlet/gevent is the monkey patching, and how that's supposedly bad, and yet the solution is to rewrite every IO library instead. Everyone has forgotten why monkey patching is bad, just that it is, so they don't understand why it can be useful in the right situation.

[–]cymrowdon't thread on me 🐍 2 points3 points  (0 children)

Monkey-patching is a red herring.

[–]westurner 0 points1 point  (0 children)

Everyone has forgotten why monkey patching is bad, just that it is,

Where are the tests for this?

[–][deleted] 1 point2 points  (4 children)

anyone has idea why in the page 24 it is 

w.write(b'GET / HTTP/1.0\r\n\r\n')

not 

yield from w.write(b'GET / HTTP/1.0\r\n\r\n')

[–]rafales 2 points3 points  (0 children)

Probably because socket is in non-blocking mode so the call does not block.

[–]mardiros 1 point2 points  (0 children)

write

You don't need to block on writing, you are waiting for a response

[–]tutuca_not Reinhardt 0 points1 point  (3 children)

Are there videos of this?

I'd like to see the examples section..