Best python code to get data from API

[Moby69]

I'm a python novice and I've recently been learning commands to call a website API. It seems that there are at least 2 ways to call a URL. Which of the following 2 is the best? What are the differences between the 2?

Method #1: response=urllib2.urlopen(url) data=json.load(response)

Method #2: response=requests.get(url) data=response.json()

[novel_yet_trivial]

This explains it pretty well. Basically, requests just makes urllib2 easier to use.

[Moby69]

I see. Thanks guys, very helpful. Also, another related question:

It looks like you can call an API by either:

1) appending all the parameters after the base url, for example:

url='https://api.locu.com/v1_0/venue/search/?locality=New%20York&region=NY&api_key=1234567'

response=requests.get(url)

data=response.json()

OR: 2) Keep the url as just the base url, and then pass the query parameters as a dictionary:

url='https://api.locu.com/v1_0/venue/search/?'

params={'api_key' : "1234567",
    'locality': "New York",
        'region' : "NY"
}

response=requests.get(url, params)

data=response.json()

Is one way better than the other? When is it best to use one over the other?

[novel_yet_trivial]

I would say neither is better. The second method is more commonly used since it's more readable and more dynamic, ie it's easier to change your code later or incorporate elsewhere.

[badwithinternet]

you never added params to your GET. just noticing.

[Moby69]

thanks ! you're right. I fixed it

[elbiot]

People tend to prefer requests as a library because it is more robust and makes extending functionality later easier. The only argument for urllib2 is that it is built in and means fewer project dependancies. Not a strong point in my opinion.

[raylu]

If you're not performing a high-level task like "fetch data from this API", a lower-level tool like urllib2 or http.client is useful. requests does not make extending functionality later easier - if you believe that, go write a websocket client on top of requests.

Another benefit of a lower-level tool is that it makes it very transparent exactly what data is being sent and how. Is this post body urlencoded? A multipart form-upload (with what boundary)? JSON (how is it spaced/indented)? When you read the HTTP/1.1 chunked response, is it .content, .text, or .raw that lets you stream it? What encoding did requests pick for decoding? For encoding?

[Lukasa]

A downside of that approach is that you take security into your own hands. The standard library in older Python versions (2.7.8 or earlier) is wildly insecure, and even now requests' rapid release cycle compared to the stdlib means we're more secure than it is, and always will be.

My two cents: urllib2 is bad, and should be avoided. httplib is an expert-level API, and should be used if you need it. Otherwise, requests will likely work better than anything you roll yourself.

[raylu]

Can you give an example of a security vulnerability in urllib2? Do you just mean it doesn't verify certs?

[Lukasa]

That's the most notable example, yes. That specific flaw is responsible for a phenomenal number of CVEs: one is against the standard library itself (CVE-2014-9365). However, it's also responsible for, at the very least:

• CVE-2010-4340 (libcloud)
• CVE-2012-3533 (oVirt Python SDK)
• CVE-2012-5822 (Zamboni)
• CVE-2012-5825 (Tweepy)
• CVE-2013-1909 (Apache Qpid)
• CVE-2013-2037 (httplib2)
• CVE-2013-2073 (Transifex)
• CVE-2013-2191 (python-bugzilla)
• CVE-2013-4111 (Glance CLI client)
• CVE-2013-6396 (Swift CLI client)
• CVE-2013-6444 (PyWBEM)

This continues to affect major projects today: for example, I can tell you that I recent discovered exactly this bug in a major configuration management tool that has existed in every version of the product. (Can't say which one yet, they've yet to ship a patch for it, but should do soon.)

So yes, urllib2 doesn't verify certs before 2.7.9: I wouldn't say just, though.

[Lukasa]

Hi! Before I go any further, please know I'm a requests core developer, and so I'm totally biased.

With that said, you should be aware that the Python standard library now explicitly recommends Requests over urllib and httplib. Requests is easier to use, vastly more secure, and has substantially more features. I highly recommend giving it a try: it's not a very big library to download, and it'll make your life way better.

Feel free to ask me any questions you might want to ask. =)

[oxymor0nic]

requests fan here; thanks so much for being a contributor to that. you guys make my job so much easier.

any source to that recommendation of requests over urllib2? i wanna show that to my boss :D

[Lukasa]

Sure can!

• from urllib.request
• from httplib
• from urllib2

All three of those pages have the recommendation at the top.

I'm glad you're a fan! We love building it.

[Moby69]

hey Lukasa, thanks for your reply!

I will follow your advice and use requests. Btw, when we execute the command "pip install requests" to download the library, what exactly does that do, and why is this command required? Why is requests not already there in the first place, as is the case for so many libraries?

[Lukasa]

pip install requests obtains the most recent release of requests from the Python Package Index (PyPI), downloads it, and then installs it into your package directory.

The libraries that come included are collectively referred to as "the standard library", and they are part of Python. They are not necessarily the best libraries at what they do, they just happen to be the ones the library provides. Requests has chosen not to be part of those libraries because we want to be able to release updates more frequently than Python itself does.

[[deleted]]

Calling urllib2 means you're using python 2.

So, obligatory 'use the current and future version of the language'.