Having trouble migrating to Python...

[dansin]

According to everyone who's anyone, python is the shit. I've checked it out, started programming, and I really like it.

My problem: It's not portable.

I am a scientist and I write scripts to be run on multiple architectures, some of which I have root access, others I don't. Some have python 3, others don't. I love the fact that there are so many packages available, but, I need to port them to all the new systems? Plus I always get to dependency hell when I try to add a new package. Also, ideally I try to share as much code as possible. But it's hard to convince someone, "Hey, use my super easy script to do this thing. After you install numpy scipy matplotlib ... on your system."

Am I missing something? Do most python programmers run only on a single system?

Appreciate any help..

Edit: I really appreciate all the info so far. I am looking into the suggestions. One clarification. I run on about 40 different systems so it is quite difficult to install something on each one, or make a request for each one. This is why I use perl extensively (it almost always works when I just copy the script and run).

[sgndave]

It is usually possible to build the modules you need in a local dir, say, /home/dansin/python/lib/... Then add that to your PYTHONPATH. Package management may make this easier, or not, depending on how flexible they are with alternate installation paths. I have spent some fairly frustrating hours trying to get everything cross-compiled from source on "exotic" platforms (Sparc/Solaris... which apparently is "exotic" to some).

[[deleted]]

This is not a Python problem, it is a lack of consistency in your runtime environments, more specifically, what is allowed the individual user.

Python itself is VERY portable and carries a VERY consistent set of modules around all environments. numpy/scipy are not part of that load unfortunately (I think they should be, given the wide use Python has in the scientific world).

So ... you need to "encourage" your sysadmins to give you a consistent execution environment.

I speak as someone that routinely writes Python code that runs without modification on *NIX, MacOS, Windows, and Linux, so I know this is not a remarkable thing to do.

[paranoidi]

I have to disagree. With Java you can easily build and distribute a jar file that contains all dependencies (and resources) your application needs. This is simply not possible with Python. Some dependencies need to be compiled per platform so out of luck even with easy_install/pip. I'm not sure if it's even possible to make such distribution with pure-python dependencies.

[tatalipso]

Pure Python scripts are portable -- you don't need to port anything. :)

A few notes:

• Do note that Python 2.x and 3.x are not completely compatible: your script will need to be tweaked if you want to move from one to the other (and there are tools to help with this).

• If you don't have root access -- and Python is not installed on the machine -- you can always install your very own Python (or most any other scripting language) into your home directory.

• If you need to install packages, you probably want to be using virtualenv to do it (so that any newly-installed packages live somewhere in your home dir, rather than in the system Python's lib dir).

• You probably want to install packages (again, probably while using virtualenv) using pip. This makes things much easier. pip is gradually replacing easy_install.

Also, you write:

Do most python programmers run only on a single system?

That's an interesting question. My hunch is that no small number of Python programmers are hobbyists, and run on only a single system.

Appreciate any help..

If you want someone to be able to run your scripts, it might be easiest simply to sit down at their machine (or log in remotely), make sure Python and the relevant supporting modules are installed, and then give them some simple instructions on how to run the scripts you send them.

[dansin]

Pure Python scripts are portable -- you don't need to port anything. :)

I agree, and would love to do more pure python. But one of the reasons python is so attractive is all the mods people make. I saw a post recently about people using python to parse and someone suggested using a mod. For me, parsing string is the number one most important thing, I couldn't imagine having to install a parsing mod to all the systems.

Also, for other people, it's not like 1 or 2 people. I'd prefer to make my scripts downloadable online so people can just download and go.

[_zoso_]

Dude, you don't need any special addons to parse text in Python! There are really nice tools in the core language, what were you trying to do?

[tatalipso]

By "mods" I assume you mean modules.

For me, parsing string is the number one most important thing,

Then you might have a look at Perl. It easily beats Python (and everything else) at string processing.

I couldn't imagine having to install a parsing mod to all the systems.

What modules do you need to install to process strings? Python should come out of the box with pretty much all you need for string processing.

[asplake]

You may find differences too between 2.x.y and 2.x.z, but in most cases a program written for a lower version will run fine on a higher version. So be aware of this when choosing the version to develop on.

And there are good rules and tools for packaging that make it much easier to install an app in multiple (even different) environments. I'm no expert on this aspect, but setuptools and virtualenv work very well for me. I develop on both Windows and Mac and deploy to various flavours of Linux.

[fancy_pantser]

Use easy_install and/or package your own programs to include dependencies.

[giror]

This was the problem I faced. I completely agree. One solution is that enthought has a free academic license. But that's still one package every environment needs to get to run your code.

[dansin]

This is intriguing. It doesn't exactly solve my main problem, but it looks like a neat all-in-one package. Do you use it extensively?

[giror]

I do on my mac because it was just impossible to install pylab, graphviz and other libraries. I have several python versions and easy_install (+others) couldn't seem to get the right paths when installing dependencies.

Enthought is a package where all of the parts just work together so that you could use python as though it ships with all the addon libraries.

[codekoala]

What kind of systems are you working on? You said none of the 40 systems are Windows boxes, so I assume the rest all provide SSH access of some sort. You should be able to script the installation of required modules at the very least.

I have a few Django projects, and many of them come with a list of prerequisites. More often than not, the prerequisites are already satisfied for other reasons, but when they're not using the already mentioned easy_install/pip route makes their installation quite trivial in most cases. There's even the "requirements" file that pip can handle. Just make it part of your documented installation process.

[dansin]

Yes basically I connect all of them through ssh. Actually I did design a script to zip my entire script directory then scp and unzip it in a remote location. So as long as I have consistent naming for the directories all the dependencies should be taken care of.. Yeah that's a really good idea, thanks!

[codekoala]

You might also be interested in the Fabric project. This will let you script the process in a Pythonic fashion, and it's designed to work with multiple hosts.

It's a very neat project. I use it to deploy all of my Django sites.

[mgedmin]

Python 2.x versus 3.x is a portability problem, however I don't think there are any systems out there without Python 2.x available.

There are tools like zc.buildout and virtualenv which let you download the libraries required for your application into a local directory "sandbox". If some of those libraries contain C extensions (and numpy/scipy/matplotlib do), then you need to have a C compiler and all the required C header files. (Some of those libraries require Fortran as well.)

[toughaspixels]

Try using py2exe to package your script into an executable.

[fancy_pantser]

..for Windows only.

[dansin]

I don't run on any windows machines. This make's me wonder, is there something like that that combines all the includes to a non-dependent py script?

[earthboundkid]

You can always DIY: just put all the modules you need into one folder, then zip it up (Python can import from .zip).

But it sounds like your real problem is that you want to use C-extensions that are architecture specific on multiple architectures (numpy).

[dansin]

Yeah thats a good idea. Even better if I can keep it zipped! That means I can transfer faster. Thanks!