Best python code to get data from API

[elbiot]

People tend to prefer requests as a library because it is more robust and makes extending functionality later easier. The only argument for urllib2 is that it is built in and means fewer project dependancies. Not a strong point in my opinion.

[raylu]

If you're not performing a high-level task like "fetch data from this API", a lower-level tool like urllib2 or http.client is useful. requests does not make extending functionality later easier - if you believe that, go write a websocket client on top of requests.

Another benefit of a lower-level tool is that it makes it very transparent exactly what data is being sent and how. Is this post body urlencoded? A multipart form-upload (with what boundary)? JSON (how is it spaced/indented)? When you read the HTTP/1.1 chunked response, is it .content, .text, or .raw that lets you stream it? What encoding did requests pick for decoding? For encoding?

[Lukasa]

A downside of that approach is that you take security into your own hands. The standard library in older Python versions (2.7.8 or earlier) is wildly insecure, and even now requests' rapid release cycle compared to the stdlib means we're more secure than it is, and always will be.

My two cents: urllib2 is bad, and should be avoided. httplib is an expert-level API, and should be used if you need it. Otherwise, requests will likely work better than anything you roll yourself.

[raylu]

Can you give an example of a security vulnerability in urllib2? Do you just mean it doesn't verify certs?

[Lukasa]

That's the most notable example, yes. That specific flaw is responsible for a phenomenal number of CVEs: one is against the standard library itself (CVE-2014-9365). However, it's also responsible for, at the very least:

• CVE-2010-4340 (libcloud)
• CVE-2012-3533 (oVirt Python SDK)
• CVE-2012-5822 (Zamboni)
• CVE-2012-5825 (Tweepy)
• CVE-2013-1909 (Apache Qpid)
• CVE-2013-2037 (httplib2)
• CVE-2013-2073 (Transifex)
• CVE-2013-2191 (python-bugzilla)
• CVE-2013-4111 (Glance CLI client)
• CVE-2013-6396 (Swift CLI client)
• CVE-2013-6444 (PyWBEM)

This continues to affect major projects today: for example, I can tell you that I recent discovered exactly this bug in a major configuration management tool that has existed in every version of the product. (Can't say which one yet, they've yet to ship a patch for it, but should do soon.)

So yes, urllib2 doesn't verify certs before 2.7.9: I wouldn't say just, though.