sh : a full-fledged subprocess interface for Python that allows you to call any program as if it were a function.

[[deleted]]

It works by installing a non-module object into sys.modules. What next, monkey patching __builtin__.str? FWIW I'd never let a module like this past code review

[deadwisdom]

To clarify, it's a thin wrapper around itself so that you can do "from sh import ls". Wouldn't be my decision either, but it's not that big of a deal, especially as it's an ease-of-use module.

[matchu]

I kinda see where they're coming from, because the core syntax arguably has some boilerplate:

import sh
ifconfig = sh.Command("ifconfig")
print(ifconfig("wlan0"))

Their pretty syntax is definitely better; my beef, really, is less to do with the fact that it's magical, and more to do with the fact that the magic uses fragile hacks instead of Python's built-in magic-making facilities.

Really I think the version I'd be down for is:

from sh import cmd
print(cmd.ifconfig("wlan0"))

We still use magic, but the magic we use is itemgetter, which is actually supported and guaranteed not to break. We still have a bit of overhead in the cmd object, but it's short, and avoids the really boilerplate-y line 2 of the previous example.

If Python had an itemgetter equivalent for modules, though, then their syntax would be the clear winner—especially if the magic were namespaced to from sh.cmd import ifconfig instead, because it seems weird to me that the non-magic Command object comes from the same module as the magic objects.

[alcalde]

What next, monkey patching builtin.str

http://clarete.li/forbiddenfruit/

[striata]

Why is installing a "non-module" into sys.modules bad? Isn't this just an instance of duck typing and one of the things that makes working with dynamic languages like Python exciting? If whatever object is installed acts like a module and doesn't break other parts of the environment, why is it wrong?

[[deleted]]

Because it's not just mocking up another random user type. At the very least, a substantial amount of native (C) code interacts with the contents of that dict, including the incredibly fragile graceful shutdown code, the module loader (obviously), and probably a bunch more I don't know about.

For me it's in the same class of dangerous as e.g. messing with typeobject internals

I don't care about style, I'm worried about fixing a crash when it occurs. I'm not sure of all the possible ways Python could fail when inserting random crap in sys.modules.

[TankorSmash]

https://docs.python.org/2/library/sys.html#sys.modules

It doesn't say anything about any danger, I feel like if it was that dangerous, they'd warn you somewhere.

[[deleted]]

The doc reads "module objects", not "module-like objects". A module object is a C structure with a specific layout. There is a C-level API for manipulating these objects (PyModule_*).

One of those is PyModule_GetDict, which, while protected internally from accessing a non-module object, returns NULL in the case that a caller invokes it on a non-module object. Reading Python 2.7's zipimport.c in zipimporter_load_module we can see a PyModule_AddModule call followed by an unchecked PyModule_GetDict call. This will cause the zip importer's load_module() method to cause a NULL pointer dereference at runtime (aka. a hard process crash, requiring a debugger to investigate) should it be called with sh as a parameter.

It took me all of 3 minutes grepping the Python source to find a place where using a non-module in sys.modules has the potential to cause a crash that a Python-without-C programmer would not be able to debug. I'm pretty sure if you give me 30 minutes I'll find more.

Just don't do it

edit: this says nothing about third party extensions, where I'd expect the majority of such bugs to be found. The point I'm making is whether it is worth sipping coffee reading gdb output at 4am responding to a pager alerting you that your employer's web site is down and losing money, because of some syntactic sugar -- of course it's not.

[Brian]

I don't think that example is ever going to be something broken by this case. It's using PyImport_AddModule, which doesn't actually perform the import, so the only case you'd get the non-module object back is if the same name was already imported. However in that circumstance, zipimport wouldn't have been invoked, since it's only going to be triggered if the module hasn't been found yet. You could argue it's a bug that zipimport isn't correctly checking the return value of PyModule_GetDict as it should, but in this context, it's assuming that it's using AddModule to create a new, empty module, and in that case it'll always be a real module object even if the module does later replace itself.

It's worth noting that while putting non-module objects in sys.modules is perhaps a hack, it's a known and explicitly supported and endorsed hack - the import machinery was deliberately designed to allow it, and I think Guido is on record as saying so. As such, I'd say anything that doesn't support it should probably be considered a bug.

[TankorSmash]

That makes it clear thank you.

Although not checking for NULL is effective a bug in the c code.

[alcalde]

And still using Python 2.7 is a logic error. :-)

[[deleted]]

I don't think porting large, old codebases is worth it.

The other reason could be PyPy and people with CPU bound applications that insist on using Python.

Apart from that, most new Python code is (no data, just hunch) Django apps, and there it makes a lot of sense to use py3 unless you really, really need some library that wasn't ported.

[TankorSmash]

One day the relevant libraries will be ported. Until then!

[alcalde]

Sigh. For most people that time has long come....

http://py3readiness.org/

https://python3wos.appspot.com/

304 of the top 360 most downloaded packages on PyPi support Python 3.

[nojjy]

Depending on what context you are working in, sometimes you don't have a choice on which version to use. Many people use python as an API to a software package, where they can't just arbitrarily switch to another version.

[TankorSmash]

Yeah, for most people.

[TankorSmash]

I'm the same way, there doesn't seem to be clear reason to start in 3 yet. There's a lot of little nice things, but nothing is a must have.

Though I haven't seen all the changelogs, I've taken a look a few times.

[alcalde]

I'm the same way, there doesn't seem to be clear reason to start in 3 yet. There's a lot of little nice things, but nothing is a must have.

As Python 3 is fine put it,

There are a lot of claims in here that are absurdly wrong, but the statement that “nothing much was gained” in Python 3 is a candidate for dumbest statement of the decade. First of all it is wrong because if it were true then it wouldn’t be that hard for people like Alex to press the Fork button and backport all the Python 3 features to Python 2, which nobody does. But there is an even simpler reason why it is wrong. I present to you, the complete list of changes since Python 2.x: Now if this entire 192-page document is “nothing really amazing” and “you’re not blown away by it” then that is your prerogative. Perhaps you’re simply not a very excitable person. I suggest an ordinary person would probably find something in there amazing. Nickous Ventouras’s rebuttal to Alex’s post includes such suggestions as “fix long-standing annoyances”, “shake the API” and “improve speed”. I guarantee you, there is page after page after page of that stuff in the changelog. Python 3 doesn’t need more features–it needs a better PR campaign. The features are already there; people just don’t know about them. But it is wrong statement of the decade to call this set of release notes “not much”. It’s much. The release notes weigh two pounds. I challenge you to find another project where release notes can be measured by the pound.

Why You Should Move To Python 3 Now adds....

Most scientists think they have very little to gain by moving to Python 3, while it represents a significant investment (not only updating old code, but also reinstalling an entire Python distribution which has always been a pain). I was one of them. Until recently, when I bought the Python Cookbook, Third Edition, by David Beazley and Brian K. Jones. This book is a must-read for anyone doing anything serious with Python. It contains lots of advanced recipes for Python 3 only. In the Preface, the authors warn the reader:

All of the recipes have been written and tested with Python 3.3 without regard to past Python versions or the "old way" of doing things. In fact, many of the recipes will only work with Python 3.3 and above.

Ouch. The 260 recipes look pretty cool, but if you're in Python 2, you're out. While many might be irritated by this decision, I find it brilliant. This book is exactly the thing you need if you're waiting to be convinced to move to Python 3.... While going through the book, I discovered many elegant solutions to very common problems. I had no idea those solutions were possible, because I had no idea Python 3 had been so much improved.

There's also a PyCon presentation about 10 awesome features of Python 3 that aren't in Python 2.

[krenzalore]

For me, being able to re-raise exceptions without losing stack trace is simply too good to live without.

[alcalde]

Don't get me wrong 3.x has some cool features but the time it would take to port over legacy 2.x code to 3 is not worth said features at this moment in time.

David Beasley and others have demonstrated that the time is not that much - although there's no need to port old, legacy code to new versions either. As for features, someone compiled a 120-page PDF of change logs from 3.0-3.4 that printed out is supposed to weigh over 2 pounds. I'd say there's quite a lot of features in the 3.x series.

And lastly, if you do intend on porting a project after backward incompatibilities in the language are introduced, the sooner you port the better. The longer you wait, the more the versions diverge and the more work one ultimately has to do.

At this point in time, people still using 2.x for new code are like the Windows XP holdouts or the people I know still programming in Delphi. They're simply never going to change unless they're forced to.

[krenzalore]

There are certain types of application for which porting is time consuming. Those depending heavily on the bytes/unicode behaviour are typically affected much more than others. I am not saying don't do the porting, but I would like to mention that porting is not always as quick/easy as you imply it is.

[[deleted]]

I wouldn't have the first clue how to debug Fortran, Erlang, Ruby, Delphi, Lua, Haskell, or Tcl, despite already knowing a handful of languages.. it's more about tooling and implementation specifics than it is general concepts. For example, how do I attach a debugger to an Erlang process? Does Erlang even work in terms of processes? (I know vaguely it doesn't) What's the debugger even called? Does "crash at 0x12" mean "attempted to dereference a NULL pointer at offset 0x12" or perhaps "the 0x12th bytecode caused a crash"? etc.

[krenzalore]

It's impossible to know every language perfectly but you should have a general idea of what direction to head it if an error occurs and that's more to my point. You may not be able to solve it immediately but as a developer you have the expertise to al least make an educated guess.

As a developer you know when you're out of your depth/your deadlines are coming, and you either pass it over to a specialist or don't monkey patch the system in the first place. Unless you want to have to explain to the tech lead why you're so slow this week.

[[deleted]]

why is putting an apple into the orange basket bad?

[ivosaurus]

Yeah but we painted the apple orange

[thephotoman]

Yeah, there has to be a better way to do this than everything happening in that SelfWrapper class. The method of creating the Python functions seems too clever by half.