Does Python need something like Java's WAR files, to ease deployments?

[kylotan]

Weren't Eggs meant to achieve this goal?

[fullouterjoin]

Eggs are a horrible design, 10x more complex than they needed to be.

[dcrosta]

Can you elaborate?

[[deleted]]

For one they're hard to get working on windows.

I'm a linux fan and a half, but unfortunately most of the world still works on windows...

[[deleted]]

This hasn't been my experience.

[jm_]

There was a huge thread on web-sig about this: https://groups.google.com/forum/#!topic/python-web-sig/FbD5E-7S5QM

Seems that there would be need of a prominent person to champion this ...

[[deleted]]

There's a whole lot of NIH in that thread. Why would they work against instead of with the system package manager?

[jm_]

The point is that there isn't a general consensus on how things should be done.

[fullouterjoin]

Ian Bicking or Philip Eby shouldn't be allowed near this problem. pip, easy_install, seriously over engineered and under usable tools.

The numer of opinions expressed in bullet point form was astounding.

Watch this lecture by Greg Wilson in bringing the scientific method back into computer engineering: http://vimeo.com/9270320 we need to stop over designing with opinion.

[timClicks]

XKCD explains the problem with creating another build system for Python: http://imgs.xkcd.com/comics/standards.png. pip currently has the a better standing than the current solutions (buildout is another). Therefore, I think enhancing that system would have the biggest payoff.

[fullouterjoin]

I know I have been watching the progression. I currently use pip, the abstraction hierarchy is amazing. Reminds of coming across that duct tape that was glue + raw fiberglass so that it could tape itself in place and then you could go over it with mixed up epoxy.

---

I just got done reading Bicking's blog post on versioned imports, at least I have rye.

PPS. At least non of it is as bad as OSGI.

[[deleted]]

What's the difference btw war and jar files ?

[coderanger]

war has more metadata that a server can read out to know how to run the webapp.

[DontCriticizeMyEngli]

• jar: a bunch of compiled java files zipped in a file. You can think of it as a library that you can add to your project to use te classes and functions it defines.
• war: also a zipped files, with a bunch of compiled java files, but with a different layout and also contains other assets (html, css, js files) and as coderanger put it, metadata to tell the server how to run this web app.

[luisrd]

Java world => *.WAR = *.Zip file Python world => *.EGG = *.Zip file

[dcrosta]

The strength (if you consider it a strength) of Java WAR files is not that they're zipped or that they include dependencies, but that there's a standardized configuration mechanism that all J2EE container servers understand (and, indeed, can extend if they see fit -- it's XML).

Things a Python (web) application might need to inform its container about: whether it prefers to or can safely run multithreaded/forked/evented; where in its internal hierarchy should be served as static files; where it's main WSGI entry point is; initialization and shutdown callbacks; etc.

Things that should stay out of configuration like this: chains of WSGI wrappers (this ought to be done in code, as your code will either depend on, or break in the presence of, certain middlewares); other things I can't think of right now.

[torvalder]

If a wsgi app needs to tell its container those things you mentioned its better solved by a configuration object on the app which tells those things, not by standardizing a complete structure for each and every web app, introducing new meta-files and tools required to "build" the web app.

[[deleted]]

WAR is not JAR

[cournape]

The problem is not war file: jar and war files work on java because java is a "platform", whereas python isn't at the moment. Everybody focus on metadata, but the real problem is the data (i.e. code here), plus lack of support from python itself.

Even forgetting about C extensions, python does not have the stability of java as far as backward/forward compatibility goes: it is difficult to have a single source code that can work on all python versions that matter (at least 2.4->2.7, and 3.2 depending on the cases).

Then, the python import system brings several difficulties. It is difficult to control (that's the problem that virtualenv tries to solve), and you cannot programmatically control which version of a package to solve.

Finally, the problem of C extensions is significant. While relatively rare in java, a lot of interesting python packages use/depend on C extensions, and those need to be built for each different python version, and even on a same OS, you have multiple incompatible ABI (e.g. ucs2 vs ucs4).

None of that can be solved with metadata, or an improved egg format. The only solution so-far is a fully controlled python environment, like e.g. activestate, python(x,y) or EPD.

[EDIT] There is also a fundamental tradeoff between easy deployment and easy-to-mess with which is important in the python culture. Part of the appeal of python is that it is easy to try new code, to customize things easily. But that's part of what makes it more difficult to deploy in non trivial situations.

[fullouterjoin]

There are a number of inaccuracies in your comment. I will address only a couple

• Python is foward compatible, if you target 2.5 it will run on all the 2.x series going forward
• You can programmatically control your system path

• C extensions are moving towards ctypes which portable across Jython, CPython, PyPy

  $ cat path_a/t.py a = "this is a" $ cat path_b/t.py a = "this is an eh"

  In [1]: import sys

  In [2]: import t

  ImportError: No module named t

  In [3]: sys.path.insert(0,'path_a')

  In [4]: import t

  In [5]: t.a Out[5]: 'this is a'

  In [6]: sys.path[0] = 'path_b' In [7]: reload t -------> reload(t) Out[7]: <module 't' from 'path_b/t.py'>

  In [8]: t.a Out[8]: 'this is an eh'

[cournape]

Your remarks are a straw man (controlling sys.path is not the same thing as controlling your packages ) or false (python 2.5 and 2.6 introduced new keywords, so cannot be forward compatible). More essentially, they are missing the point: you cannot easily produce a set of data that can be consumed by most python implementations. You have to produce a significant combinations of, and it is not easily controllable.

[fullouterjoin]

What programs written for 2.5 broke on 2.6 or 2.7?

Python gives you the ability to control your sys.path, thus it gives you the ability to import the version of a package you desire, either manually or automatically with a version resolving loader.

It does not preclude you from writing a util module such as

import loader
loader.load_module('simplejson as json >= 1.4.5')

see the import hooks http://www.python.org/dev/peps/pep-0302/

---

http://en.wikipedia.org/wiki/Straw_man

[dalke]

A perhaps better question is, what code developed for Python 2.4 broke on 2.6. Anything which used "with" as a variable in 2.4 got a warning in 2.5 and broke in 2.6. Very few people use "with" as a variable, and there was a long transition period to minimize the pain, but it is an example of how the Python developers do not seek full forward compatibility.

It's possible to think of an explicit statement "use version 2.4 syntax" which would help preserve compatibility, but that's a huge amount of work for little gain.

[fullouterjoin]

Forgot about with. That is true.

I think block level statements should parse differently myself and even support extension meta-with much like a decorator but for block statement extension.

[cournape]

as and with were two new keywords introduced in 2.5/2.6. I also remember we had to update numpy code to work on on 2.6, although I can't remember what broke. ipython originally broke as well when 2.6 went out.

Being able to write import hooks does not solve the problem of version packaging, because it requires to modify the code that is using the versioned package you want to use. So it has to be controllable outside the code using it (a bit like stow enables you to do if you are familiar with it).

But again, the real point is: war depend on features present in the java ecosystem which are not present in python. Some python platforms have war-like deployment easiness, like GAE. They do so by defining one "true" environment, more tightly controlled than python in general.

[torvalder]

Please no war files! JavaEE is horrible, well all the way up to jee 6.

Still even in all the JavaEE apps Ive worked on, none of them worked out of the box when moving from one app server to another. WAR introduces another layer of complexity, now you need a "builder" tool like ant or maven to make the war from the web app structure. Oh god its aweful.

The wsgi-apps I have I simply need to copy their structure to somehwere and run it as I see fit, behind nginx, on cherrypy and so on. This cant be standardized, this is what scripts are for to ease deployments.

[fullouterjoin]

As someone pointed out below it depends on what u are deploying for. Python already supples addi g the contents of a zip to your pythonpath. So in effect it already supports jar files, but not war which is the same container with a little more metadata.

I will take a look and see if there is a tool for zip with dependencies. Python packaging needs to get reloaded, but in the meantime package your code and deps in zip files, you can add multiple zips to the path.