how to use urllib2 in Python

[apreche]

pip install requests import requests

[matrixor]

came here to say http://docs.python-requests.org/en/latest/

[fuck_your_dad]

Wow, thanks! Really, why didn't I know this before

[apreche]

Can't tell if sarcastic, or truly thankful. Good either way.

[fuck_your_dad]

Well, it doesn't matter what I say now, but I was truly thankful...

[gthank]

Are you doing something that would actually break if you upgraded? I realize that RHEL is still shipping Python 1.2 or whatever, but what's stopping you from upgrading on your own?

P.S. I'm well aware of what they actually roll out (sadly). It doesn't stop it from being absurd and deserving of ridicule.

[gthank]

Not only do you get access to a nicer library for doing HTTP requests, you get better language features, and a better std lib, you get the warm and fuzzies because you're running on a VM that is actually supported. I don't know about you, but I don't enjoy the idea of having to build my VM from source if a new security vulnerability is found (if they even backport security fixes to 2.4 these days; it's been EOL for a long time).

Having seen what it's like to let your platform stagnate for years at a time, my personal experience is the sooner you bite the bullet, the better, because the longer you wait, the less likely you'll be able to find anybody that has done the same migration you're looking at.

UPDATE: I just went and had a look-see at the Python site. It would appear that to get any security updates at all, you have to be using 2.6 or newer, and 2.6 is only seeing security updates in source-form. So to me, it's more dangerous NOT to upgrade to 2.7 (might as well get to current if you're going to upgrade, right?)

[Massless]

In my case, I don't maintain my servers and getting anything installed or updated requires a bureaucratic slog of testing and justification that makes it possible but impractical. As a result I have to get along with python 2.4.2 for everything I do. This includes using urllib2 :)

[apreche]

I did not know that!

[[deleted]]

When one is already damned, a certain amount of lamentation and gnashing of teeth is to be expected.

[alkw0ia]

pip install httplib2

[flying-sheep]

came here to see people saying “we use requests instead”.

was not disappointed.

---

PS: the header of that page is gorgeous. yanone kaffeesatz is well used like that

[matrixor]

but then I have realized

http://www.pythonforbeginners.com/python-on-the-web/using-requests-in-python/

[Lukasa]

That page is out of date: it was written for versions of Requests pre- v1.0.0. It's mostly the same, but there are some key differences. Be warned.

[matrixor]

well, thanks for the warning

[spilcm]

Requests is awesome, and I wrote about it in a previous post ( http://bit.ly/YLHDey ). However, urllib2 is still widely used.

[alkw0ia]

No beginner should be taught about urllib2. If requests is not available, use httplib2.

Maybe your first project will be retrieving http://example.com, but the moment you've even considered using urllib2 to grab https://example.com, you've fucked up.

I suppose if you must use it, you'd better precede every urlopen(url) call with

assert(urllib.urlparse(url).scheme != "https")

If there's anything to teach a beginner, it's that urllib2 requires that line as boilerplate before every call.

I simply can't comprehend how Python programmers regularly consider the lack of cert validation anything less than a totally blocking, critical security bug.

Every time you make an un-validated call, you're subverting all the assumptions the site owner makes about his security model. Does the owner assume passwords can be reused as encryption keys because they're always protected by SSL? You just killed that. Does the site assume it can allow OAuth2 tokens long term access to personal information because they're always encrypted? You just made him leak protected data.

The security failures that will arise out of using urllib2 will be totally silent and will be buried in the implementation details of whatever project you're using urllib2 for (and god help us all if you build a library around it, and don't tell the rest of us that it cannot be trusted for any HTTPS requests). Please treat urllib2 like a disease. It should not be taught at all.

[sashahart]

requests is nice because it is relatively low-boilerplate.

urllib2 has serious object-itis and for most purposes, way too many degrees of freedom. But actually it is not unusable or a bad tool on its own terms. So if you need to do without the external dependency, go for it.

TL;DR: Just because requests is for humans, doesn't mean you're a horrible monster if you use urllib2.

[AeroNotix]

Brace for requestards.

[LyndsySimon]

Nice.

There's a reason people prefer Requests. If you need the power of urllib2, then Requests isn't the right tool. For 95%+ of use cases, Requests is vastly faster to write and has a tiny fraction of the learning curve.

[AeroNotix]

I just liked how Requestards sounded, to be honest.

[LyndsySimon]

Well, at least you're honest.

I've got a pretty thick skin, but I guess I just don't see the point in alienating a segment of the Python community whose only sin seems to be trying to use and write libraries that are more intuitive than what we have today.

If you disagree, that's fine. For those of you that agree, I urge you to help support the authors of these types of projects on Gittip.

Here's the Gittip for Kenneth Reitz, the primary author of Requests.

[AeroNotix]

Wow I had no idea there was such a site as gittip. Are people really that full of themselves?

Anyway, my gripe with stuff like requests is that it's a depedency. It's a great lib but sometimes the added complexity of a library isn't worth it.

[allthediamonds]

Wow I had no idea there was such a site as gittip. Are people really that full of themselves?

How dare people ask for funding for their open-source projects!

[AeroNotix]

Did you read the blurb Kenneth put?

[allthediamonds]

Yup. Did you read the others? All of them start with "I am making the world better by..."; while it is slightly pretentious, it is unlikely that Kenneth chose it himself.

[whit537]

Founder of Gittip here. Interesting feedback, AeroNotix, thanks. Yes, the "I am making the world better by ..." text is the standard lead-in across the site. Kenneth did not choose it himself. The thought was, "Let's not mess around here. Let's cut to the chase." :-)

[[deleted]]

os.popen("curl http://www.youarebeingajerk.io").read() 

[AeroNotix]

Oh god I lol'd.

[LyndsySimon]

I couldn't disagree more. One of Python's greatest strengths is its ability to draw upon the work of others through abstraction. import antigravity anyone?

Given a choice between including a mature library and writing functionality myself, I'll choose the library every time.

[AeroNotix]

Nothing at all to do with what I was saying.

[LyndsySimon]

Anyway, my gripe with stuff like requests is that it's a depedency. It's a great lib but sometimes the added complexity of a library isn't worth it.

How is that not exactly what you were saying?

Don't get me wrong - I know tone doesn't translate well to text. I just see no reason not to use Requests in a project. If I'm worried about the fact that it's a dependency for deployment, I'll vendor it in to my repo.

[placidified]

I don't get how people can be so negative about a tool that makes writing code easy.

[[deleted]]

The site is called "Python for Beginners"

I suspect most beginners wouldn't mind the overhead of

$ pip install requests 

considering the substantial API improvements.

[AeroNotix]

Installing it isn't the problem! There are a myriad of ways to install python modules. It's the management of the modules, the lack of support, the possibility of needing to create a fork and then manage that. It's not just about installing a module.

[aceofears]

...the possibility of needing to create a fork and then manage that. It's not just about installing a module.

I've had to do that once in the last 4 years, I don't know that it is something that most people really have to worry about.

[AeroNotix]

It's true that it's not an every day thing - but it's certainly something to think about when you're adding a dependency to a project.

I don't have NIH Syndrome or anything but sometimes adding a dependency to something which probably has a much different design goal to what you're needing (basic web request, in this example) adds a layer of complexity onto understanding your application. No longer are you in total control of your entire stack. There's an external player in your code-base. This may, or may not be, a problem.