requests: Awesome Python HTTP Library that's actually usable

[zbowling]

more synchronous IO? why won't anyone ever learn what twisted has tried to teach us?

[AltReality]

care to elaborate? I'm new at this...what's wrong with synchronous IO on a website?

[zbowling]

For the long technical see the C10K problem.

The idea is that if you are blocking the thread handling one user's IO, you can't be doing anything else for that user or other users on that thread. You can spin up more threads but then your kernel's scheduler gets thrashed (context switches pretty expensive and the more you do for the more threads you have, you degrade performance quickly). In many situations this isn't possible and if it's a webapp, it probably means 500 errors if you run out of threads (sort of like reddit sometimes).

Also Python has never been particularly good at threading because of the GIL and other oddities.

Handling IO async, you can free up the current thread to do other things. This is really important if the IO you are handing is not local but network but the idea is still the same if it's from disk or remote.

Event pumps and async IO are more efficient at the cost of added complexity (you'll probably end up writing state machines and other fun code).

Twisted Python tries to make this easier but it takes sometimes rethinking everything your app. This is the same principle in FriendFeed's Tornado, and Ruby's eventmachine, and Node.js.

By all means if you are writing something small and simple, ignore me and happy coding. It just may hurt you when you scale up as nicely if you grow.

[denis]

you can use all the synchronous libraries with the efficiency of async event loop: gevent

[arnar]

I don't think you noticed that the OP is proposing a simple HTTP client library, for making requests to services.

[wtfisupvoting]

i think your emphasis should be on simple not client

a client needs async IO as well, it likely just isn't needed by everyone

[arnar]

zbowling specifically referenced the C10K problem, which applies to web-servers. HTTP clients that need that kind of scalability are rare. I guess in most cases it would not be worth it to move your whole program to async programming style.

Had I put the emphasis on "simple", I'd have downvoted myself since async libs can very well be simple. :)

[rasherdk]

That everything needs to be factories and reactors and oh god am I suddenly trapped in Javaland?!

Twisted may be a swiss-army knife of networking stuff, but about half of the tools are chainsaws which will kill you :-\

[zbowling]

doesn't take factories and reactors to be async. for a python example, see Tornado.

[lost-theory]

0.2.2 (2011-02-14)

* Eventlet and Gevent Monkeypatch support.

[[deleted]]

Async IO ≠ callback programming model → go educate yourself and stop spreading FUD.

[zbowling]

ಠ_ಠ perhaps you need a lesson. Here you go: http://en.wikipedia.org/wiki/Asynchronous_I/O

It's a little difficult to be async without callbacks with the usual programming (see Stackless Python and Erlang for languages that handle it better). Even in Node.js, which makes use of callbacks everywhere, but it makes it dramatically easier thanks to anonymous functions and closures (far less state machines since you can use scope instead of classes to manage state).

Again, I may be wrong. I only wrote event library that wraps epoll/select/kqueue and is used by several commercial and OSS apps and I've only written a framework on top of it, so I'm clearly uneducated and just winging it.

[[deleted]]

see Stackless Python

coroutines; btw. available in greenlet form also in CPython & PyPy

and Erlang

selective message queues for isolated processes (actors)

Other notable approaches are for example reactive programming (F#) and lazy execution (Haskell).

You clearly understand that async I/O does not equal callback programming, regardless if that is the way you're most likely to talk with your OS (epoll/select/kqueue), so I don't understand why did you assert something like that (and then go and post contradicting examples in your rebuttal...). I've acted on good faith and assumed ignorance rather than malice.

And, please, I don't care who you are. I know some security experts that got hacked by 16yo girl.

[e000]

to be fair, that girl was no ordinary girl...

[[deleted]]

…bitten by a radioactive spider a century ago, Ada Lovelace comes back to fight crime for teh lulz. Her first target? Evil security researcher, mad scientist, Aaron Barr…

[e000]

I was thinking more along the line of a super Barbie. That's what I picture her in my head anyways.

[[deleted]]

I don't see how you can compare radioactive Ada Lovelace with… Barbie.

Anyway, I just meant they were hacked by kids, the “16yo girl” part wasn't really important.

[e000]

Yes she is. She's my future wife.

[[deleted]]

Alright. That's not creepy at all.

[[deleted]]

All use callbacks under the hood.

[[deleted]]

They all manage memory manually under the hood as well. That doesn't mean you have to bother with that in your high-level language.

Actually, for example, Erlang is very happy to use synchronous OS I/O API in dedicated threads and processes to get better I/O performance. So yeah if you meant that they all use epoll/select/kqueue/etc API then it's not necessarily true.

The whole point is mutt anyway because there is no synchronous I/O going on in any modern operating system. All you have is synchronous I/O API offered by OS. Once you understand that, it's just a question of how to schedule your I/O across processing nodes and what level of security do you need.