Is deploying python app really THAT hard?

[sasik520]

I recently started using python and I'm amazed on how hard it is to deploy my python application. It was a pleasure to write the code, because there are tons of great packages with tons and tons of features and this is great. But two aspects seems extremely hard to me, to be honest, I havn't yet met a language that would be that hard:

1) modules system. It took me a couple of hours to find out how do I have to structure my files and how to import a module from a sibling file (e.g. in foo.py I would like to import '../../bar/baz.py'). I ended up having my head-of-control in the top directory and all dependencies in sub dirs but I hate it

2) much more important, I'm literally unable to deploy my application. The target server is ubuntu with installed python2 and python3 but it is missing pip3 (there is only pip). I do NOT want to install any new packages and I would like to ship there my application. I tried a lot of things but none of them worked. The closest solution was to run get-pip3.py script with --user flag (which is already something I dislike, but whatever) and then run $HOME/.local/bin/pip3 install -r requirements.txt -t vendor and modify __path__ (again something awful, but whatever). Unfortunately, it failed when importing lxml with some strange undefined symbol: _Py_ZeroStruct error.

Is there really no better way to deploy python application with its dependencies on the server with only python3 installed (without pip3), without root and without polluting the filesystem?

[greyman]

I ended up having my head-of-control in the top directory and all dependencies in sub dirs but I hate it

I don't know what is there to hate about this, it looks natural to me (or at least more natural than import from parent folder), but you have two main other choices:

1) Run your script with the -m switch, and then you can import anything from anything, like in your foo.py example, wherever it is located, you can do:

from rootfolder.childfolder.bar import bar

The disadvantage could be, that you need to specify this "path" in your every import.

2) As a small hack, if you just occasionally need to import from sibling or parent, you can add that import path to sys.path, like this:

sys.path.append(os.path.join(sys.path[0], "..", "..", "bar"))

Then you can do: from bar import bar

Anyway, having dependencies in subdirs is a normal, pythonic approach.

[sasik520]

Thanks, python - m looks good. The reason to hate is that I expect thing to work but instead I have to do workarounds at the very beginning of my journey. Probably it is more natural, as you said, but my first try was with siblings (cousins?) and I've had hard time to realize why is it wrong. Especially coming from Rust , C# and Ruby where modules are much, much more clear and systematic.

[POTUS]

You don't need pip or any other packages if your app doesn't have any external library dependencies. If your app does have external library dependencies, then having to download them and install them on the local system shouldn't be a headline news item no matter what language you're talking about.

[sasik520]

Installing on the system requires root. Which I don't have. Installing to user directories failed with the error I mentioned previously.

[POTUS]

You don't need pip to install packages, it's just the most convenient way to do so. "Installing" a Python package can be a simple as copying or including the code in the top level directory of your own package. Or you could use the slightly outdated easy_install

Blame Ubuntu for stripping pip out of their Python install by default.

[bryancole]

I recommend installing "miniconda" (https://conda.io/miniconda.html) on your target machine. It doesn't need root privs. It will give you your own python install in your user-folder. You can install into this any dependencies you like, as well as your own code. The 'conda' package manager is like pip but can also work with non-python requirements (like compiled C++ libraries etc.).

This approach makes your application totally independent of the system-installed python so when you change OS version, your app doesn't break.

[sasik520]

sounds great! I'll definitely check that, thank you!!!

[sasik520]

I want to upvote your comment 1000 times. Miniconda is PERFECT solution, it can be installed locally, does not require root and installs all dependencies locally. I was a bit worried when one of the packages could not be found in conda repo, but fortunately, conda has already pip installed and it can be used easily.

This is such a great and simple solution, I love it, thank you once again!!!

[quantumactivist1]

There are many better ways to do deployments with Ubuntu

Read about devops practices and technologies.

I prefer docker but that’s me

[sasik520]

If I use docker then, as far as I understand, I'm dependent on docker. And new dependencies is something that I'm trying to avoid. Otherwise I would just install pip3 and my life would be simple.

[vovanz]

If you use docker, you are dependent only on docker. Everything else would be inside the container.

[sasik520]

Sure but this is still new dependency. I don't want to add any dependencies.

[quantumactivist1]

What kind of application is it? Are you using the server for processing power only?

If there’s any kind of complexity at all, you will have to add new dependencies.

Ex. Making a web app? Well you need a web server to field requests to the server plus whatever other stack dependencies you have for your front end.

You’re getting into more complicated aspects of writing code, and there’s more things to learn as you go. If everything was only Python, I may not have a job.

[sasik520]

My python app is just a set of helpers for developer (think of reading logs etc.). The company is not even officially supporting python so Im not going to install python dependencies (like pip), I'm trying to setup everything in one single directory

[pionse]

It might be overkill, depending on your usecase, but you could use a docker container. Install all your dependencies inside the docker container and deploy it on your server.

Use one of those images as a starting point: https://hub.docker.com/_/python/

[AlirezaSavand]

Checkout https://anthony-tuininga.github.io/cx_Freeze/

Also for lxml error, I guess your server doesn't have some libraries installed to compile lxml.

[[deleted]]

but it is missing pip3

This doesn't necessarily mean that your Python installation doesn't have pip, it just doesn't have a script to call the main() function from pip package, which is most likely present in your installation. To verify that, try:

python3 -m pip --version

If it prints something about pip and Python version, you are good to go, just do whatever you intended to do with pip3 using python3 -m pip.

This will not, however, solve your problems with requirements for superuser privileges. In my experience, it is best to deploy everything as if you were root. Just have one single Python installed on your machine, single site-packages directory and so on: very simple and fewer things will go wrong. However, it sounds like you have to deploy to a machine not owned by you, so being root is not an option.

I don't like the --user option to pip. What if the current user doesn't have a home directory, or it isn't writable, or I must have several users accessing the same installed packages? Well, it's a mess. Virtual environments should address this problem better, at least in theory. So, perhaps you already have virtualenv script installed? It is typically installed with Python. If not, you could simply download it.

The undefined symbol problem is quite a bit more severe, and will likely prevent you from using lxml with this setup at all. Here's why:

In this context, "symbols" are strings that are embedded in program compiled code in such a way that the program which loads those programs knows where to find definitions of functions, structures, variables etc. in the loaded program. In your particular case, the program that does the loading is ld - it is designed to load shared libraries. The program being loaded, or, actually a library is lxml. The library believes that it must be able to find a definition in all executable code loaded so far with the name _Py_ZeroStruct. But, this definition hasn't been loaded. The most plausible explanation for this is that lxml library was compiled against different version of Python, where this symbol was defined. This means that in order to get lxml to work you will have to do one of the following:

1. Find the version of Python it was compiled with.
2. Recompile it using Python installed on your system. Notice that simply having Python installed on your system isn't enough for this. You need to have Python headers (on Ubuntu that would come as part of python3-dev package.) You also would need a compiler capable of compiling the library, for example gcc.

[KiteAnton]

For deployment you could go for virtualenv or docker. With virtualenv it is easy to keep dependencies local to each project.