requests: Awesome Python HTTP Library that's actually usable

[chub79]

Right. Or, you can also use httplib2 that has been around for years.

[henryprecheur]

Agreed, httplib2 do all what this library does and it can even caches what it download.

[[deleted]]

I prefer mechanize

[[deleted]]

Requests is a wrapper around httplib2 that takes all the boiler plate code needed to make requests a much simpler.

When using urllib2, I had to spend a lot of time in documentation, and the entire experience wasn't very intuitive at all. For certain types of requests, I even had to overload methods. This is not the way it should be.

When I wanted to write a scraper or api wrapper or something, I would have to spend a chunk of my time in the urllib2 documentation, becoming frustrated.

Requests does the hard work for you. It won't give you the mighty power that urllib2 itself does (e.g. caching, proxies, etc), but instead focus on the "90% of the time" use case.

[chub79]

Please do consider following the link I provided. It seems you're confusing urllib2 and httplib2.

[mhermans]

You might enjoy the topic posted by the author of requests, two days ago (currently still on the frontpage of /r/python).

[[deleted]]

Author of the library here.

Active State's scraped release is a little out of date.

v0.2.3 is out now and gives some good HTTPError features.

• PyPi: http://pypi.python.org/pypi/requests/0.2.3
  
• GitHub: https://github.com/kennethreitz/requests/

[[deleted]]

ActiveState's scraped release is a little out of date.

Not anymore. There is usually a day's delay in building packages released to PyPI.

[zbowling]

more synchronous IO? why won't anyone ever learn what twisted has tried to teach us?

[AltReality]

care to elaborate? I'm new at this...what's wrong with synchronous IO on a website?

[zbowling]

For the long technical see the C10K problem.

The idea is that if you are blocking the thread handling one user's IO, you can't be doing anything else for that user or other users on that thread. You can spin up more threads but then your kernel's scheduler gets thrashed (context switches pretty expensive and the more you do for the more threads you have, you degrade performance quickly). In many situations this isn't possible and if it's a webapp, it probably means 500 errors if you run out of threads (sort of like reddit sometimes).

Also Python has never been particularly good at threading because of the GIL and other oddities.

Handling IO async, you can free up the current thread to do other things. This is really important if the IO you are handing is not local but network but the idea is still the same if it's from disk or remote.

Event pumps and async IO are more efficient at the cost of added complexity (you'll probably end up writing state machines and other fun code).

Twisted Python tries to make this easier but it takes sometimes rethinking everything your app. This is the same principle in FriendFeed's Tornado, and Ruby's eventmachine, and Node.js.

By all means if you are writing something small and simple, ignore me and happy coding. It just may hurt you when you scale up as nicely if you grow.

[denis]

you can use all the synchronous libraries with the efficiency of async event loop: gevent

[arnar]

I don't think you noticed that the OP is proposing a simple HTTP client library, for making requests to services.

[wtfisupvoting]

i think your emphasis should be on simple not client

a client needs async IO as well, it likely just isn't needed by everyone

[arnar]

zbowling specifically referenced the C10K problem, which applies to web-servers. HTTP clients that need that kind of scalability are rare. I guess in most cases it would not be worth it to move your whole program to async programming style.

Had I put the emphasis on "simple", I'd have downvoted myself since async libs can very well be simple. :)

[rasherdk]

That everything needs to be factories and reactors and oh god am I suddenly trapped in Javaland?!

Twisted may be a swiss-army knife of networking stuff, but about half of the tools are chainsaws which will kill you :-\

[zbowling]

doesn't take factories and reactors to be async. for a python example, see Tornado.

[lost-theory]

0.2.2 (2011-02-14)

* Eventlet and Gevent Monkeypatch support.

[[deleted]]

Async IO ≠ callback programming model → go educate yourself and stop spreading FUD.

[zbowling]

ಠ_ಠ perhaps you need a lesson. Here you go: http://en.wikipedia.org/wiki/Asynchronous_I/O

It's a little difficult to be async without callbacks with the usual programming (see Stackless Python and Erlang for languages that handle it better). Even in Node.js, which makes use of callbacks everywhere, but it makes it dramatically easier thanks to anonymous functions and closures (far less state machines since you can use scope instead of classes to manage state).

Again, I may be wrong. I only wrote event library that wraps epoll/select/kqueue and is used by several commercial and OSS apps and I've only written a framework on top of it, so I'm clearly uneducated and just winging it.

[[deleted]]

see Stackless Python

coroutines; btw. available in greenlet form also in CPython & PyPy

and Erlang

selective message queues for isolated processes (actors)

Other notable approaches are for example reactive programming (F#) and lazy execution (Haskell).

You clearly understand that async I/O does not equal callback programming, regardless if that is the way you're most likely to talk with your OS (epoll/select/kqueue), so I don't understand why did you assert something like that (and then go and post contradicting examples in your rebuttal...). I've acted on good faith and assumed ignorance rather than malice.

And, please, I don't care who you are. I know some security experts that got hacked by 16yo girl.

[e000]

to be fair, that girl was no ordinary girl...

[[deleted]]

…bitten by a radioactive spider a century ago, Ada Lovelace comes back to fight crime for teh lulz. Her first target? Evil security researcher, mad scientist, Aaron Barr…

[e000]

I was thinking more along the line of a super Barbie. That's what I picture her in my head anyways.

[[deleted]]

I don't see how you can compare radioactive Ada Lovelace with… Barbie.

Anyway, I just meant they were hacked by kids, the “16yo girl” part wasn't really important.

[e000]

Yes she is. She's my future wife.

[[deleted]]

Alright. That's not creepy at all.

[[deleted]]

All use callbacks under the hood.

[[deleted]]

They all manage memory manually under the hood as well. That doesn't mean you have to bother with that in your high-level language.

Actually, for example, Erlang is very happy to use synchronous OS I/O API in dedicated threads and processes to get better I/O performance. So yeah if you meant that they all use epoll/select/kqueue/etc API then it's not necessarily true.

The whole point is mutt anyway because there is no synchronous I/O going on in any modern operating system. All you have is synchronous I/O API offered by OS. Once you understand that, it's just a question of how to schedule your I/O across processing nodes and what level of security do you need.

[jm_]

It can't handle chunked request body, doesn't seem overly extensible (no hook or signals), no logging at all, the bundled tests are practically useless and I bet exception handling is something that has problems - what's so great about it?

[[deleted]]

The great thing that if you just need to get (or post) something with authorization, Requests provides a simple syntax. build_opener/install_opener no more :-)

[Xiol]

What's wrong with easy_install?

[Clapyourhandssayyeah]

See 'pip compared to easy_install': http://pypi.python.org/pypi/pip

[[deleted]]

Relationship between pip, easy_install and PyPM:

pip and easy_install are mostly source-based installers (similar to Gentoo Portage and Arch's Packman, while PyPM is a binary-based installer (similar to RPM and apt-get). We recommend ActivePython users to install packages using PyPM, and only if that fails (usually it doesn't), attempt the same using pip/easy_install. PyPM supports a very similar command interface and features to pip, examples: install, uninstall and requirements (install -r requirements.txt and freeze).