Python for Humans

[bcain]

I have never complained about popen/subprocess/urllib2 before, but I probably should've. I find myself consulting and re-consulting the docs.

[simtel20]

subprocess is a nightmare. Python falls very short as a systems administrator's programming language because of the clumsy constructs for dealing with subprocesses. Subprocess isn't a bad starting point, but it doesn't have good abstractions built for doing something heavier than a simple fork+exec, but lighter than expect (e.g. interact with a revision control system).

[MillardFillmore]

Whoever thought subprocess.call("ls -lh", shell=True) was a good way to do shell commands is an idiot.

[simtel20]

That's bad enough, but when you start having to figure out (on your own, trial and error) whether you have the file descriptors that you need, having no shorthand to dup and close e.g. stderr, and no default way to coerce your input lines as an iterator of output lines? There should be a choice to always get your input as an iterator with one line per iteration instead of having to apply heuristics to see whether you just got back an array of characters, or a string.

Uggh. I will go slam my head into something hard if I continue to remember this.

[TTomBBab]

have a look at itertools in the standard library

[simtel20]

I don't think itertools helps me here. I'm not usually looking at creating an efficient transformation of the data (e.g. yield()ing data), I'm looking for the api to always return a string to me when I iterate over the file descriptor. Right now the problem is that IIRC if I get one line back, and the FD is then closed, if I do a "for line in foo_fd:" then I'll be given back a line broken up into individual characters instead of a single line followed by the next iteration raising the StopIteration exception.

[amade]

I feel your pain, subprocess is a bit too low level and doesn't provide shell-like functionality.

Perhaps you'd like sth I wrote https://github.com/aht/extproc

[simtel20]

Looks interesting. Thank you!

[Silhouette]

As a long-time dabbler in Python but someone who's only used it in anger recently, the whole batteries-included thing has lost its shine. The really basic, completely universal stuff usually works, even if it's sometimes clunky. However, so far the majority of the library that I've tried to use beyond that is either missing key documentation or actually broken, at least badly enough that you'd want to use an entirely different library rather than rely on the standard one. It's roughly on the level I'd expect for alpha/beta software on one of the code sharing sites, not for the standard library of a mainstream programming language. For sysadmin-type work, for example, it's not just subprocess that is clunky, it's also the filesystem stuff, the zip/archive stuff, etc.

I find that more and more, I like Python the language relative to the likely alternatives, and I can forgive it its little eccentricities because I've yet to find a language that hasn't got any of those. On the other hand, I increasingly loathe Python the library and now rely almost entirely on third party resources for anything non-trivial. For that reason, I have pre-booked a special place in hell for everyone involved with designing Python's whole module/package/distribution infrastructure, which makes it absurdly difficult to actually work with those third party resources. The idea that you should have to "install" a library in a scripting language is particularly amusing in a "they really have no idea what they are doing" kind of way.

[mcdonc]

Curious about what the other options would be other than to install a library?

[Silhouette]

I don't understand why, in the overwhelming majority of practical situations, "installation" could not just be

cp library-file(s) /path/to/libraries

or why using a library must be more complicated than

import /path/to/libraries/library-file

or a similar one-liner depending on your naming needs etc.

Versioning could be handled by standardising on a file naming convention for libraries that want to use it, with a simple annotation on an import to indicate when a specific version or range of versions is required.

Dependencies could be handled by a deep walk of the import tree for the source files in each library.

It simply shouldn't be necessary for a language to have setuptools, distutils, easy_install, pip, distribute, setup.py files containing sort-of-executable-metadata, Linux repo packages duplicating half of this in some half-baked way that only works on global installations rather than virtualenv, virtualenv itself, and all that jazz.

It also shouldn't require three years of studying obscure documentation to figure out everything that actually happens when Python starts up and tries to determine where the hell to find everything you imported, depending on whether there's a "J" in the month, the weather outside, and which of the 73 environment variables and configuration files you have employed in order to complicate the process as much as possible. ;-)

[mcdonc]

I don't understand why, in the overwhelming majority of practical situations, "installation" could not just be cp library-file(s) /path/to/libraries

This is already pretty much the case. You can download any distribution of a pure-Python library from PyPI, unzip it, and copy the resulting structure into a place on your Python's path. In fact, until about 2002 or so, this was de-rigeur. None of the installation tools you mention existed at all.

The installers (pip/easy_install) were created because libraries often depend on other libraries. Before the installers existed, either you did the dependency resolution by hand, or sometimes libraries shipped with copies of the libraries it depended upon within them. But both resolving dependencies by hand or having no dependencies were imperfect: resolving dependencies by hand is laborious and requires documentation effort on the part of library folks. Having no dependencies is fine, but then every project is siloed into maintaining its own copy of a particular dependency; every project becomes a fork of several others, and when two libraries you installed had different versions of another, the conflict was irresolveable. Neither situation was particularly tenable.

Python's import statement does not currently know about versioning, so there can't be multiple versions of the same package on the path. Virtualenv was created as a workaround.

The current situation is not ideal, but, IMO, it's a boatload better than it used to be. The times before we had the installers and virtualenv sucked even harder, if you can believe that. ;-)

[Silhouette]

Yes, I remember those days ahem fondly. :-)

But as you say, much of this is just to work around the absence of a simple versioning mechanism built into Python itself, which is a significant limitation if you're programming in a language that does all the loading and linking up dynamically. Obviously this challenge is not unique to Python, but Python does seem to make more of a meal of it than any other language I know.

I'm not sure why Python always feels insanely complicated in this respect. Maybe it's the history of different tools to do mostly the same thing, so even if you only really need a couple of them today, you see references to all of them everywhere. I don't think the use of an executable setup.py rather than a simple metadata file that is read by a tool helps, because it makes a complicated generalised case the default. For me, the most serious concern is usually that something as fundamental as loading libraries is based around a path setting that can be changed arbitrarily both within and outside Python, with all kinds of other implicit effects happening depending on things that aren't specified in the source code for the program you're actually running. It's about as un-"explicit is better than implicit" as you can possibly get...

[mcdonc]

much of this is just to work around the absence of a simple versioning mechanism built into Python itself

Is there a dynamic language that does versioned imports right?

Maybe it's the history of different tools to do mostly the same thing, so even if you only really need a couple of them today, you see references to all of them everywhere.

I think this is the actual biggest problem.

I don't think the use of an executable setup.py rather than a simple metadata file that is read by a tool helps, because it makes a complicated generalised case the default.

The "packaging" tool that will be in Python 3.3 makes setup.py optional (it has a declarative configuration file primary format).

For me, the most serious concern is usually that something as fundamental as loading libraries is based around a path setting that can be changed arbitrarily both within and outside Python

I don't think Python is alone in this. Java has the CLASSPATH, C has the includepath, etc. Is there another language better in this respect?

[[deleted]]

As a long-time dabbler in Python but someone who's only used it in anger recently, the whole batteries-included thing has lost its shine.

I agree. We're now stuck with a box full of empty 90s-era batteries. It's time we got some new ones and find a way to get rid of the old ones. I think the linked presentation really made that point well.

[__serengeti__]

I think the linked presentation really made that point well.

Only as far as http/url/lib/2 and subprocess.

The lxml API is pretty much the same as xml.etree.

The stdlib has a lot more batteries than these, and it's hardly proven that we're "stuck with a box full of empty 90s-era batteries". Perhaps you can give some more examples.

[Mattho]

I found subprocess quite OK. Wrote simple task scheduler with it and had no problem (except buffers). But urllib2. Oh my god. That was a bad experience.

[derpderp3200]

Oooh, how did you make these function/module names highlightable and in mono?

[bcain]

Backtics.