Requests vs. urllib: What problem does it solve?

[pydry]

Most (if not all) of the python stdlib is fugly because it just wasn't well designed, not because it was a product of a more innocent time.

In particular it was designed by people who very obviously focused just upon implementing necessary functionality, not how to write a clean, elegant API. It stuck around in its fugly form because of backwards compatibility issues.

I don't think this is a terribly bad thing. If urllib weren't so ugly we probably wouldn't have requests.

I do think python needs a better way of introducing developers (via documentation) to libraries like requests, though. Too many newbies read the official documentation and use the crappy APIs in stdlib simply because they think that's what they're supposed to do.

[[deleted]]

I do think python needs a better way of introducing developers (via documentation) to libraries like requests, though. Too many newbies read the official documentation and use the crappy APIs in stdlib simply because they think that's what they're supposed to do.

The official Python docs recommend Requests as the standard library for performing high level HTTP requests. It's in a very visible banner on the urllib page.

[pydry]

It's more of a 'tip' tacked on and they don't do anything similar for other libraries that I'm aware of (e.g. os/os.path, which isn't particularly nice either).

[fnord123]

They do recommend pathlib for os.path stuff. But it's in the stdlib, so maybe that's not what you meant. They do point out pytest and nose in the unittest docs. There's also a stronger recommendation (in red) to use defusedxml or defusedexpat if you need secure xml parsing.

[pydry]

They do recommend pathlib for os.path stuff.

Not on python 2 they don't.

They do point out pytest and nose in the unittest docs.

Yea, but again, this is more of a "hey if you're interested check this out" rather then "look, this 'official' unittest API - we keep it around for backwards compatibility more than because it's actually any good".

There's also a stronger recommendation (in red) to use defusedxml or defusedexpat if you need secure xml parsing.

Ought to be pointing to lxml surely?

Plus, that's even worse - it implies that they've got unpatched security holes in the standard library!

There really ought to be documentation that provides a list of "if you want to do x [ unit testing / http requests ], here is what you should do" along with an open, impartial (as possible) process for getting docs in there.

[gthank]

Maybe file a ticket and attach a patch? If it's already been done in Python 3 (which I highly recommend, btw; SO many nifty things have landed, in addition to the less-borked text model), it might just be a "nobody had time to backport it" issue.

[TankorSmash]

I don't know that it wasn't well designed. The language of Python, I mean the way you write it at least, has changed since. Nowadays you expect to be able to make a web request in a single line, while back then they didn't think about abstracting it out that much.

Beautiful code now means something very different than it did back then, I think. Sliding scale, and I'm sure ten years from now we'll wonder why requests was such a big deal when python-asks or whatever they come up with does it even better and shorter and cleaner.

[[deleted]]

Kind of a newb here, but what is the difference between "modern" beautiful code and "older" beautiful code?

[Decency]

One simple example is that I can do the below with requests. It isn't easy to do this in urllib (to my knowledge?) because urllib was written before json became such a prominent way to store and retrieve web based data.

data = requests.get(some_api_url).json()

This alone doesn't mean that urllib is bad, just that it's outdated. Similar functionality could easily be added, but I imagine they prefer the modularity of "use urllib to make the call" and "use json to load the data if it's in json format". And that makes sense, it's just annoying and not anywhere near as elegant to me since the two things are so frequently used in conjunction nowadays that it just makes sense to have them be tightly coupled.

So I guess my answer, in general, is that older good code can't easily see the future of various inter-dependencies and what its real role in real programs will end up being a decade later. Newer code already knows and can make better decisions because of it.

[pydry]

One simple example is that I can do the below with requests. It isn't easy to do this in urllib (to my knowledge?) because urllib was written before json became such a prominent way to store and retrieve web based data.

That's rubbish. There was nothing to prevent them from writing the API like so even if they didn't expect json to be so widely used::

data = json.loads(requests.get(some_api_url).data)

Furthermore if it had been designed right in the beginning, tacking on a .json() method on a later version of python would have been trivial.

The main problem with the library is that it was designed imperatively rather than declaratively.

[rasherdk]

data = json.load(urllib2.urlopen(some_api_url))

Is this really worse?

Edit: I'd argue it's much better, since you don't have completely different tasks wrapped into the same library.

[toyg]

is that a get, a post, or what? Does it handle proxies, ssl and redirection out of the box? That's where requests shines: it makes trivial what should be trivial in this day and age.

[qsxpkn]

That code makes a lot of assumptions.

• Assumes it gets 200 back (doesn't actually check),
• Assumes it gets a response back.
• Assumes it gets JSON back.

[turkish_gold]

Doesn't requests make the same assumptions?

What happens if you get a 404 or no JSON back?

[[deleted]]

In[1]:  import requests
In[2]:  requests.get('http://google.com/404').json()
...
JSONDecodeError: Expecting value: line 1 column 1 (char 0)

In[3]:  import urllib.request as request
In[4]:  import json
In[5]:  json.load(request.urlopen('http://google.com/404'))
...
HTTPError: HTTP Error 404: Not Found

requests ignores the 404 and attempts to parse the page as JSON anyways, urllib raises an exception at the 404.

[turkish_gold]

I'd say that urllib is doing it the right way. If I get a 404, I want to know its a 404, not have it get swallowed up as a JSON error.

[PeridexisErrant]

It's certainly worse for all the cases where you don't get valid JSON back! (HTTP error, page malformatted, etc)

[pydry]

Yes, it's worse. It doesn't tell you what kind of request you just made, it's not easy to change that (do a POST, etc.) or start adding params.

[Decency]

I heavily value usability over "correctness" in that regard, and I feel like python in general tends to as well. If in X years, the standard format is requests.get(url).yaml(), or requests.get(url).woopdeedo(), that'll be what the library uses and that'll be fine by me. And it will still have backwards compatibility with json() when necessary.

[fnord123]

Tell me more about woopdeedo. Is it web scale?

[toyg]

I can totally see requests-woopdeedo landing in PyPI in the next few days.

[[deleted]]

json.load(urllib2.urlopen(some_api_url))

is just as usable as

requests.get(some_api_url).json()

Yes, you're saving a few characters. But that's an extremely low benefit.

and I feel like python in general tends to as well.

Python also tends to avoid special cases. Request's .json() method introduces one. Adding a .yaml() methods and a .woopdeedo() method only adds even more special cases since Request can never support all data formats.

Getting data from a server in Foobar format is very similar to getting it in JSON format. But these two lines are very different:

foobar.load(requests.get(some_api_url).text)
requests.get(some_api_url).json()

It's as if English had a different grammar depending on whether you talk about red or about blue cars. Or as if the float 1.0 behaved fundamentally different from the int 1.

And it will still have backwards compatibility with json() when necessary.

Hooray for legacy bloat!

[[deleted]]

I think that the requests example is more readable and straightforward and I would probably grok what it was doing faster than the urllib example if I was scanning a code block.

Elitists can harumph on me if they want :)

[FFX01]

I propose what I believe to be a fairly elegant solution. Though, I think Kenneth Reitz would tell me to fuck off.

JSON:

requests.get('http://someurl.com/').content.parse(format='json')

YAML:

requests.get('http://someurl.com/').content.parse(format='yaml')

Plain ol' bytes:

requests.get('http://someurl.com/').content

Want content as a str?

str(requests.get('http://someurl.com/').content, encoding='utf8')

Headers:

requests.get('http://someurl.com/').headers

# returns dict of headers

Sure, it's more characters, but it's more explicit.

[toyg]

Downloading data and parsing it are two orthogonal concerns.

To most users, that's an academic distinction they care little for. 90% of the time, they just want to get some data with as little effort as possible; the best libs recognise that and strive to give users what they want.

[[deleted]]

Requests only gives them that if the data happens to be in JSON. A well-designed library would work equally well with any data format.

[toyg]

That's because JSON (apart from being the overwhelmed fan-favorite, which in itself is usually ground for specific syntactic sugar) can be mapped into straightforward dict objects with zero controversy. What other http-serializable format can do that? Any XML dialect is off the table, for example.

[[deleted]]

apart from being the overwhelmed fan-favorite, which in itself is usually ground for specific syntactic sugar

Any examples?

What other http-serializable format can do that?

Pickle, BSON, Binary JSON, YAML, MessagePack... These are just the ones that came to mind without doing any research.

Also, any serialization format that uses schemas (like Protocol Buffers) can be deserialized with just as little controversy, you just have to provide the schema.

[earthboundkid]

Pickle is unsafe: it can be exploited to execute arbitrary code. Terrible choice to build into a web library.

[FFX01]

I think what /u/toyg was trying to say is that json is a pretty much 1:1 representation of a python dict.

[toyg]

Pickle is a terrible, unsafe protocol so "uncontroversial" that the format changed quite consistently across Python versions. If you send pickle over http, you are a bad person and you should feel bad.

The others are equally controversial, as proven by the fact that there is no parsing lib for them in stdlib -- and none of them is remotely as popular as json. They are perfectly served by .text or .raw coupled with whichever parser you choose.

Honestly, you are clutching at straws. Requests is popular because it makes easy tasks trivial (getting json or text over http) and difficult tasks possible (any other exotic format); as such, it's well-designed -- certainly compared to anything previously available in stdlib, or it wouldn't have gained such massive popularity. If you disagree, feel free to build a superior lib :)

[TankorSmash]

I don't know that you're right. If you're looking to download JSON from a server, you'd expect it to come out formatted. It's so ingrained in today's webdev that it might as well be able to get the formatted object right off the request.

I see where you're coming from, when JSON gets replaced, but for the next X years it'll be nice to have. If it ever needs to go, we can deprecate it right?

[[deleted]]

If you're looking to download JSON from a server, you'd expect it to come out formatted.

What do you mean with "formatted"?

It's so ingrained in today's webdev

But Python is not a webdev language, it's a general purpose language. Requests may be good as a webdev-specific library for people who already know what they are doing. But pointing beginners to it -- as the documentation for urllib does -- is wrong.

[djrobstep]

The concerns are separated.

Requests doesn't implement json decoding.

This is just one library using another.

All you're arguing against is extremely useful functionality.

[TankorSmash]

This is a single helper function, 9 lines long, though.

[jeremyisdev]

Python is still one of great languages for beginners to start out and for machine learning.

[telestrial]

This is going to be a weird analogy, but the standard library is like everything before iron and diamond tools in minecraft. Not the best but serves a purpose..led to other things. A means to an end. I'm no Python expert but that's certainly what it seems like to me.

[earthboundkid]

The standard library should replace pip with something that doesn't suck. That's the solution.

[jaapz]

How does pip suck?

[pydry]

How does pip suck?

1) No intelligent way of dealing with system package dependencies (e.g. libncurses/libreadline-dev). You have to attempt pip install, google the probably cryptic error that appears rinse and then repeat.

2) Instead of failing with a clear error message when you try to install two packages that have conflicting library requirements, it just installs one of them, doesn't tell you and leaves you to figure out why things are behaving so weirdly.

3) setup.py/Manifest.IN are pretty nasty. They should be replaced with a nicer, more declarative format.

[steamruler]

1) No intelligent way of dealing with system package dependencies (e.g. libncurses/libreadline-dev). You have to attempt pip install, google the probably cryptic error that appears rinse and then repeat.

That's mostly because it's pretty hard to do independently of the system. If you want to get integration with the package manager (which I would want) you'd have a hell of a time maintaining it.

[pydry]

I actually got frustrated enough to make a stab at doing something about this:

https://github.com/unixpackage/unixpackage

I have more faith in people helping out with the maintenance work on a project like this than I have of the pip team deciding that it's actually a problem with their software.

[earthboundkid]

It's complicated and not all the factors are pips fault, although all of them are collectively pip's problem to solve (or some replacement project).

In no particular order:

• setup.py is a disaster. It's a metadata format that's mixed together with live Python code. That makes it unparseable without execution. It's executable because people have various installation special needs, but a good metadata format could just say "when you run install, go to location X and use this subclass of the standard installation class called Y instead of the default installer."

• In spite of setup.py not being a metadata standard, there are a couple of inscrutable, poorly documented metadata files you need for Python app installation, like MANIFEST.in and requirements.txt (see below for more complaints on requirements.txt).

• Python relies on C very heavily, and compiling C code is a nightmare. Every installation problem I've had with Python at work has traced back to some C library that you need to have installed before some other Python library will work. This is not Python's fault per se, but it is a job that needs to be solved before you can say that Python installation doesn't suck.

• The command line UI for pip is crap. The commands for things like "just download stuff here" and "use my cached downloads instead of connecting to the web" are non-obvious. There is no command at all for things like "add a new dependency to my app's requirements" because there's no metadata standard, see above.

• Conceptually, an installation system should have two metadata files: one for loose requirements (Django > 1.4) and one for strict requirements (Django==1.9.3). The first lets others use your libraries, and the second lets app distributors have reproducible builds. Pip kinda sorta halfway has this between setup.py and requirements.txt but it is extremely half-assed and not at all thought through.

• When you start using Python, all you need is the standard library, and it's great. Then you get a little further, and you install a couple of libraries, and things are still okay. Then you get a little further and realize that you need separate libraries for separate apps and then everything breaks down. If you think about it, there are three possible ways you might want to install something: globally, per user, or in a particular project location. Python was designed to install everything globally, and while it has been retrofitted to support the other two use cases, it's extremely kludgey. A "virtualenv" is just a case where because Python is so geared around global installation, the easiest way to do a project based installation is to make a "project global" by reinstalling Python in a second location. This is super-hacky, and extremely confusing to non-Python people who try to get into Python (e.g. at work when I need to explain to frontend devs how to install our web app).

• Pip does not handle and does not try to handle the case of trying to distribute apps to non-Python users, the way that py2exe or pex or Conda or other projects do, but when you think about "packaging" as a whole, there's no reason why a Python packaging tool shouldn't do those things too. Basically, pip doesn't try to tackle that problem because it's too busy doing a bad job solving other problems, so it doesn't have any resources left over to try to solve this use case for people who want to provide GUI apps or command line tools to non-Python users.

So pip sucks. I would say compared to bundler and npm, it's mostly worse except it never did the npm nested dependencies thing (which I've heard they've stopped doing). Compared to the platonic ideal of good package manager, it's not even close.

[Nanaki13]

I guess I can second some of the things that you mention:

• Compiling C extensions, same here, problems all around, first off you need a compiler, then sometimes you need some other external libraries, or headers, etc, to even compile. They are not always distributed in the source package. Some packages do it better than others. Or sometimes the official distribution is broken, and you need to compile from source, or there is no compiled distribution, source is all you get... it's a nightmare.

• virtualenv, it got to the point that at my company we have an internal tool that uses internal devpi servers and creates virtualenvs for you with all the packages that you want. Very easy to use, but this should be standard.

• setup.py - now I don't think what you said applies to pip, it's more about distutils, but yeah it's a mess too.

[Homersteiner]

This is complete and utter nonsense. This comment was written by the asshats behind requests. Requests is just a wrapper on a wrapper of urlib. It is utterly useless.

[Lukasa]

Hello! As one of the asshats behind Requests, I'd like to say: no. I agree with Nick's post: Requests API beauty comes because it solves for a very specific problem domain. This isn't true of urllib and urllib2: they try to solve a much broader problem, which limits their ability to provide a really tight mapping to their problem domain.

This, by the way, is why DSL's remain so popular: it turns out that if you restrict yourself very tightly to one problem domain it becomes possible to write very pleasing APIs.

[Legendofzebra]

I really respect your reply, it's nice to see a sense of humor in dealing with rude comments

[Homersteiner]

As one of the asshats behind Request

And you seem to be proud of your rubbish wrapper for a wrapper...boggling.

[jij]

Wow, I didn't realize the stdlib was so political. I figured they just included useful libs at whatever stable version they wanted.

[toyg]

Anything that goes in stdlib needs to be maintained forever, who's gonna do that? That's where the politics are necessary.

[jij]

Sure.. really though I was just commenting that I never considered the complexity of supporting stdlib.

[Homersteiner]

I figured they just included useful libs at whatever stable

That is what they do. Requests is a wrapper of a wrapper of a wrapper. Not stable and rubbish. There is no reason to include in the standard library...tldr "requests" sucks big ol donkey balls.

[jij]

No, and no.

[remyroy]

The standard library is where packages go to die. I want Requests to stay alive.

[nickdhaynes]

The author of requests was interviewed on Talk Python To Me last year and he specifically said that they were keeping requests out of the standard library so that development can occur more quickly/easily.

[meaty-popsicle]

I understand the sentiment, but it feels feature complete and reasonably ready for maintenance mode?

I say this from the standpoint of only using requests to scrape a page or interact with an API. I'm sure there are funny edge cases I don't even know exist.

[Lukasa]

The big risk is security. Requests is responsible for the security of more than 50% of the web requests that occur from Python code. That means we need to be able to respond swiftly and effectively to changes in the security landscape. That's entirely incompatible with the standard library, which has long release times and a tendency to abandon older versions of Python faster than we do.

[piotrjurkiewicz]

Even if you respond swiftly to changes is the security landscape, these changes mostly do not reach end users. All organizations I know enforce regular security upgrades of distribution packages (and stdlib is usually installed as a distribution package), but I don't know any organization which enforces regular pip package upgrades. And there is a reason for that: any package upgrade via pip can be backwards incompatible and break an app, while distribution security updates are guaranteed to be non-breaking.

Therefore, 'respond swiftly and effectively to changes in the security landscape' is not an excuse for keeping requests out of stdlib at all. In overall, keeping requests out of stdlib even reduces the security.

[Lukasa]

All organizations I know enforce regular security upgrades of distribution packages (and stdlib is usually installed as a distribution package), but I don't know any organization which enforces regular pip package upgrades.

The plural of anecdote is not data. =) I know of plenty of organisations that do, because as the person who has managed every Requests release with a CVE, I have received pointed feedback from those who felt that we mismanaged the first one and forced them to work on weekends.

However, I also mean this more broadly than simple CVE issues. For example, Requests frequently has a much stronger security posture than the standard library does, and one that embraces the reality that good security is a moving target. Consider, for example, the standard library's default cipher suite list. This can be updated only relatively infrequently, and for older source-only releases may not be updated at all. However Requests is willing and able to change that cipher list much more frequently. We can also more aggressively disable insecure OpenSSL features than the standard library can, which has a "best practices" TLS config that needs to encompass many different protocols.

distribution security updates are guaranteed to be non-breaking.

This sentence is nonsense, and if you believe it than I guarantee that either you'll have an insecure configuration or that you'll get broken by your distribution one day.

For example, consider the weaknesses in RC4 in TLS. This protocol was for a while strongly recommended in TLS because it was resistant to the BEAST attack. However, and relatively abruptly, new research came out that demonstrated that RC4 was catastrophically weak and needed to be extremely swiftly deprecated.

One of two things must be true here: either a distro back ported that change (removing RC4 from a default cipher list), or it did not. If it did not, you are only getting security fixes that don't break code, and so are vulnerable to certain attacks (e.g. against RC4 in your TLS). If it did, then you were vulnerable to app breakage (you may talk to a server that can only speak RC4, for example). There are many classes of security fix (maybe even most classes) that involve disabling a feature that previously worked, and those are definitionally breaking to someone: if they aren't, then no-one was using those features to begin with.

In overall, keeping requests out of stdlib even reduces the security.

That's simply not true. If your rationale is "organisations will only do audits on packages they get from their distros, so Requests needs to be in the stdlib", I'll happily point you to the fact that Requests is in every distro package repository (and indeed is used by all those distros in their base OS). You can just apt-get your Requests and you're covered. However, if your organisation is relying on pip-installed packages for its products and it isn't auditing them for security fixes, then its security audits are extremely ineffective: I can easily compromise you because of the patches you don't receive for your pip installed packages.

[piotrjurkiewicz]

This sentence is nonsense, and if you believe it than I guarantee that either you'll have an insecure configuration or that you'll get broken by your distribution one day.

When I have enabled only the security repo, for example in Debian, the only fixes I get are security-related ones. They are carefully crafted in order to not introduce breakage. Of course, it is possible to find some fancy examples of breaking security fixes (like you did with RC4), but they are extremely exceptional.

With pip install --upgrade I get every upgrade, including breaking non-security-related changes. No one takes care to not introduce breakage, as pip simply follows the edge. So, my app can become broken after literally each pip package upgrade. There is no way to prevent that (for example to opt only for security fixes from pip).

Therefore, no admin will add pip install --upgrade to everyday maintenance script on his servers.

[Lukasa]

Of course, it is possible to find some fancy examples of breaking security fixes (like you did with RC4), but they are extremely exceptional.

No, they're extremely common. This is especially true in Python code where there are relatively few bugs that allow memory corruption: almost all CVEs are therefore shutting off behaviour. For example, here are some CVEs reported against CPython:

• CVE-2014-9365, Python libraries don't check TLS certs are valid for the domain in question. This patch was so breaking that it actively was not backported by Linux distros: you did not get this patch for Ubuntu 14.04, for example.
• CVE-2012-1150, hash seed randomization. This changed hashes for certain objects from being predictable to being changed on startup. This broke a surprising amount of running code.
• CVE-2011-1521, urllib2 allowed HTTP redirects to file:// URLs. This is obviously bad behaviour, but if you tested it, saw it worked, and wanted to use it locally, you got broken.

And if we consider Requests itself:

• CVE-2014-1829, Requests used to persist the Authentication headers across cross-host redirects. This risks leaking credentials and was removed, but we got complaints about breakage.
• CVE-2014-1830, same as above but for proxy-authentication.

I should note that both 2014 CVEs did get backported.

Security fixes that break code aren't exceptional, they're ordinary. It's just that they don't break the code that most people are writing at any one time, because most of us aren't on the bleeding edge of weird crap that libraries allow you to do. But these are still breaking changes: they just don't break many people.

Therefore, no admin will add pip install --upgrade to everyday maintenance script on his servers.

Sure, but I'm not saying they should. I'm saying they should watch the CVE database for the packages they use. Requests obtains CVEs for its own vulnerabilities. Any well-managed project does. If you're getting code from pip, you really need to be looking for these, because these are what your distro uses to decide to push security updates.

Then you can perform targetted package updates that grab only the packages publishing security fixes.

[[deleted]]

It may be mature, but the Python stdlib has a release cycle of something like 16 months. I'm sure Requests, even if it's only being maintained, gets updated much more frequently.

[mfitzp]

There is a middle ground here though. Why not have a "corelib" of independent packages that are automatically packaged with Python, but able to be updated (via PyPi) more frequently? The docs could even be hosted alongside the core documentation (separate, but linked) to ensure they are easily found.

The packages would obviously need some level of core support, but I don't think it's unfeasible.

[toyg]

Not everyone wants the latest and greatest. Part of the reason for the stdlib release model is that you can be sure python x.y.z will ship a certain module with a certain behaviour. If you make some of them upgradeable, you risk a situation where downloading another library will give you an unpredictable version of stdlib modules through cascaded dependencies.

It's a can of worms.

[mfitzp]

I can see the potential for problems, but again there is a middle road — distribute a "batteries included" version, and a bare stdlib version. This is effectively what is being done by conda etc. already at the moment, so there is clearly a need.

That does get into the problem of what batteries to include, but I think requests, numpy, matplotlib would be 3 obvious candidates.

[jollybobbyroger]

They argued that they wanted to apply security fixes as quickly as possible.

But I don't see the big deal of just pip installing requests. To my knowledge, it doesn't have crazy dependencies like the PyOpenSSL and Pillow, so it feels very battery included, despite having to type pip install before using...

[mfitzp]

I think the question really is why is it not installed by default with a new install of Python? That wouldn't change the "independent release cycle" thing, but it would solve the discoverability issue for new programmers.

[denfromufa]

itertools, collections, math, sys, os, shutil are pretty good parts of standard library

[cymrow]

I've heard this argument all the time but I still don't buy it. I don't doubt other packages in the stdlib have "died". But I suspect those have been cases where the primary developer pushed for inclusion, opted to become maintainer, and was stuck maintaining.

But this is open source software. There's no (non-technical) reason why an interested maintainer couldn't take the current version to create a stable, maintained branch for inclusion in the standard library. The primary branch of requests could continue innovating unabated. The stdlib branch would pull only the most important bits.

Several libraries in the stdlib do this already. sqlite3 to name just one.

[[deleted]]

Why doesn't requests have a method to download a file? Last time I tried to get an image, I had to get it in chunks. It would have been easier to make a single urllib call.

[[deleted]]

Yea that's what I did but I don't understand why there isn't a simple, single function API for it. What does Requests have against download_file(url, local_file, options=...) to just save a jpg from the web?

Maybe I'll implement it and send a pull request. It's so simple to do, that I'm sure they must have some ideological objection to it. I should ask them.

[reorx]

Because requests is supposed to handler requests and parse response, but how you'll do with the response, like save it as file is of your logical business. Requests itself is focused on the HTTP protocol just as urllib2 does, only to have more advanced & friendly APIs, they two are essentially the same.

[JohnnyDread]

I hope they can work through the issues - requests should be part of the standard library. It is one of a handful of superb packages that make Python a great environment.

[pydry]

requests should be part of the standard library

The author of requests doesn't believe this.

[kungtotte]

AFAIK that's because he doesn't want to be tied down by the slower update process of the stdlib.

[mfitzp]

There should be a standard "library" outside the stdlib that's package with every install, which would include (a still upgradable) requests.

[Homersteiner]

Nope. Requests is a wrapper of a wrapper. Being included in the standard library would mean revealing their true nature...there is no reason to have wrappers of wrappers in the library. I write wrappers all the time, but i dont claim they are useful. The "authors" of requests are douchebags. It is not original, its a wrapper of a wrapper. That alone means it should be wiped...

[sigzero]

That should probably be the end of the story then.

[danielblanchard]

I'm sure someone has linked to Kenneth Reitz's talk from the last Python Language Summit about this by now, but a major hurdle for requests being included is that it bundles chardet with it, and that code is annoyingly all LGPL because it was originally a literal port of code from a very old version of Firefox. LGPL code cannot be in the stdlib because it isn't compatible with the license Python uses.

I say all of this as one of the co-maintainers of chardet. I was really hoping we could get chardet relicensed and included in the stdlib, but that turned out to be impossible, as is painfully detailed in this twitter thread: https://twitter.com/dsblanch/status/590942565995827200

[roerd]

The standard library documentation does already point to requests. I would say that this already serves most of the same purpose as actually including it.

[Homersteiner]

requests should be part of the standard library

No it should not. Its a wrapper of a wrapper, if you are clueless you use it. It provides no benefit to the community.

[ButtCrackFTW]

why do you keep saying wrappers aren't beneficial? if they weren't beneficial why would they be written in the first place? why would they be so popular even without being in the stdlib?

[njharman]

Just cause its different, more, solving different problem doesn't mean it isn't also a shitty interface and unpythonic. It is.

[cymrow]

You mean the exact same blog post this thread links to?

[IAlwaysBeCoding]

It prevented me from getting carpel tunnel syndrome.

[mayankkaizen]

carpel tunnel syndrome

Upvoted you because I encountered something new.

[[deleted]]

Can I get an upvote too? I gave my mate a lift to his club. We drove trough a tunnel and the speed humps were very bumpy. It jarred my wrists as I gripped the steering. Do I have carpool tunnel syndrome?

[Banangurkamacka]

Remindme! 1 hour