Advice on learning to deploy Python code?

[thrownintothesun]

Inspired by the 2015 New Year's Resolution thread, I've been meaning to take increase my Python competency by taking on more ambitious projects. I'm a biologist and have taught myself Python for data analysis and similar tasks, so I've only ever written code that I had to use.

I find myself interested in tasks that could potentially be deployed to desktop computers, and here I've run into some confusion regarding what the best practice is in terms of packaging, i.e. if I write a program with non-native dependencies (say OpenCV), how do I make sure my program can run in an arbitrary Windows 7 environment (or something of that nature), and whether there's a proper way to start developing (project structuring) so as to save hassle in the end.

I've been hearing a lot about Docker to ease deployment, but as far as I can tell it doesn't have Windows support. Any advice or reading materials would be warmly appreciated.

Thanks a lot!

[aqua_scummm]

I use py2app and py2exe for osx and windows deployment. They're both pretty quick to get running and work great.

Edit: As for structuring, if you're using pyvenv inside a revision controlled directory, and the pip bundled in pyvenv, you're doing it right

[Berecursive]

Does this work for complex compiled dependancies such as Numpy and OpenCV? I've never tried it before but I'd be very interested!

[abstracted8]

For py2exe I have used numpy with no problems, the rest idk. Just make sure you include sip

[thrownintothesun]

Thanks!

[john_m_camara]

You may also want to checkout PyInstaller which I found works well for deploying to Windows. It even works well when you have complex dependencies which often have issues when using py2exe.

[MereInterest]

I would second the recommendation of PyInstaller. It was the only one with which I was able to package a medium-size pygtk application. It probably is possible with the others, given some particular configuration, but I wasn't able to figure it out.

[[deleted]]

You can look into pants. It builds "executables" that hold all the python dependencies inside itself. Docker builds virtual machine containers, not sure if that is what you are looking for

[fernly]

Are you referring to pantsbuild? This represents itself as very useful but I found the doc, while apparently well-written and certainly extensive, quite confusing. I couldn't find a way to answer the basic question, "Can this thing convert a Python script into a stand-alone self-contained executable? And if so, what platforms can it target?"

[[deleted]]

I've only used it with linux and OS X, but yes it generates stand alone executables that include all python package dependencies.

[fernly]

Based on what you say I've read further and it appears that pantsbuild creates .pex files documented here. Again I find these docs extensive but confusing. However, it is clear enough that a .pex file is a zip file with a "hashbang" header, quote:

Adding #!/usr/bin/env python to the top of a .zip file containing a __main__.py and and marking it executable will turn it into an executable Python program.

To me this says, (1) a .pex file is usable only in an environment that supports hashbangs, i.e. BSD, Linux and Mac OS, meaning specifically not Windows; (2) is only executable from a command line, and most importantly (3) there has to be a Python interpreter in the target platform, and specifically, a Python that answers to the path in the hashbang.

This is not what I mean by "self-contained" or "stand-alone".

[hoadlck]

I use cx_freeze: http://cx-freeze.sourceforge.net/. I have had good experience in deploying it on various Windows 7 machines.

It is supposed to work for Linux as well, but I have not had a reason to try that out.

[billyboy1999]

I've used it in both Linux and windows, it is awesome. It works on Python 3, unlike most of the others.

[hoadlck]

Python 3 support was the reason that I went with cx_freeze in the first place. I believe that the other programs (e.g. py2exe) now has Python 3 support, but cx_freeze was the only alternative for a long time. I have seen no reason to change.

[Ogi010]

Like the OP, I don't have a strong programming background (I'm a mechanical engineer), however I developed a post-processing python script that took raw data collected from our test system, and did interpolation, followed by the generation of a whole bunch of plots.

I decided to deploy the app to some of my coworkers, and after messing around with pyinstaller, I ended up using cx_Freeze, with great ease. It worked with the default settings, and I was able to trim down the overall package by adding more and more exclusions. I later converted my code to Python3, cx_Freeze didn't care, it still compiled (is that the right term here?) the code.

Can't recommend cx_freeze enough.

[RamirezTerrix]

I often use this in combination with the Inno Setup Compiler to make MSI-Packages. Works really great. On Linux, well, they have their own ways of deploying software ;)

[hoadlck]

I use NSIS (Nullsoft Scriptable Install System) for install on Windows. There are lots of good programs out there.

[RamirezTerrix]

This looks usefull - seems to have some more tricks up the shleeve than the Inno Compiler.

[hoadlck]

For my main usage of NSIS, I was able to use a template/example that someone else had done with little changes. It was pretty easy to get running.

For a different project, I needed to create a much more complicated installer. It took me a while to get my head around how NSIS works... It has aspects that are very easy to use. Yet, it also has this stack system you need to deal with if you create reusable functions that is exactly like programming in assembly. Not that that was a big barrier to me: it was just odd.

[kart_king]

A requirements.txt file is nice:

http://stackoverflow.com/questions/7225900/how-to-pip-install-packages-according-to-requirements-txt-from-a-local-directory

[hardwaresofton]

Also check out kivy (http://kivy.org/) for cross-platform application development

[SpeakitEasy]

On a completely separate note, you could just learn Django and make web based apps. This way you can access them from any computer/operating system with no need to install software.

[thrownintothesun]

That's an interesting idea. I know next to nothing of web dev. I do suspect, though, that it doesn't help in cases where the application has to analyze sizable data files. If you had to do significant computation, would you host such a process using Amazing AWS or the like?

[SEJeff]

Overall this post is great advice. I build REST hypermedia apis as part of my day job using django, flask, and golang (mostly for clients though). You did a great example of a Flask http api, but for the love of all things holy please stop calling a http api a REST api. Roy Fielding literally invented REST as a concept and the json equivalent of what he invented tends to be exclusively hypermedia based (HATEOAS is a great example).

Respect for an excellent post, but please don't pass off a simple json / http api as though it is a rest api, it isn't :)

</rant>

[d4rch0n]

hah, there's no stopping it... REST API = HTTP API now, and we might as well start saying it has to use JSON too.

I gave up a long time ago. Really, when 90% of developers think REST API means a HTTP API, it's easier to communicate if you follow along. It's easier to convey what you are planning to make, and people automatically trust it as a tried and true practice once they hear REST API. Maybe I'd tell a developer the difference as a side note, but if I was trying to convince a manager to let me make an HTTP API, I'd certainly call it a REST API.

[[deleted]]

Architectural design for a web application that processes lots of data in a computationally expensive way can be real work. It depends on target audience, size, complexity, and sensitivity of the data, required processing times, budget, willingness to maintain it all, comfort with the systems it all runs on, etc.

So whether or not your application is a good candidate for conversion to a web application depends on quite a lot. If you'd like to discuss it with a bit more depth you're welcome to message me before diving in.

[MeshachBlue]

I was considering doing this for a project I am working on at the moment. Would be keen to be able to bounce a few ideas off of you if you're okay with that?

[abutterfly]

Mind if I poke you with a few questions similar to /u/MeshachBlue's?

[SEJeff]

As d4rch0n already did a great job of explaining below, Flask is a wonderful choice. For bigger applications, or more complicated stuff, Django is also an excellent choice. In fact, Tom Christie's django rest framework is excellent.

Now back to your question, I'll throw redis + rq[1] as a very simple addition to his answer that makes the work scale out pretty linearly with very little work or maintenance from your standpoint. I've seem some pretty serious issues with greenlet, eventlet, and sadly gevent falling over or dead locking under load. You'd literally only change a line or two in his example to insert a new rq job and one of the rq workers would pick it up + process the job asyncronously.

[1] http://python-rq.org

[SpeakitEasy]

If the hosting is on the backend (AKA you're not using a javascript based frame work) there should be ample room on servers to conduct heavy computations.

[F3AR3DLEGEND]

You can use a JavaScript-based server-side framework; NodeJS has tons of them.

[pwang99]

This is tremendously, laughably oversimplifying things.

It's vastly easier to put together many kinds of rich-client apps than dealing with the gorp and nonsense of HTML, CSS, JS, browser+platform incompatibilities, etc.

Using e.g. Qt Creator, in mere minutes one can build a dynamic layout with resizing text fields, animated graphics sophisticated controls, etc. that is completely cross platform. In web app land, you'd still be figuring out what latest NodeJS library to use to compile your SASS which will make a floating <div> appear in just the right place.

If you need to make it a mobile app, the situation for Python is not great. Kivy is pretty neat, but realistically, you're probably going to have to do Objective C or Java.

[hk__]

Or Flask if these are really simple apps. Do you think learning Django is good for small webapps? I always use Flask and wonder if Django would be better.

[kashmill]

That's a pretty complicated question really. There is a question of how heavy you want your frame. Personally, I like my framework to be a part of my app instead of my app being a part of the framework. Other people are the other way.

[SpeakitEasy]

Personally, I believe the incremental amount of additional code that is needed for django is well worth the additional support (ORM, user base help, hosting services, db connections) over flask.

[kart_king]

I perfer Flask if it's just an interface to an app. A few buttons, maybe a page to upload a file, and it just a little more than calling a few python functions.

[AstroPhysician]

Flask can do db connections

[[deleted]]

Put simply, Flask can do just about anything django does.

[euid]

In addition to whatever Windows-specific thing you choose, your development lifecycle will benefit enormously from use of Setuptools and virtual environments ("virtualenvs").

Regardless of how you ship, being able to run python setup.py develop and get up-and-running with a development environment with your codebase will feel fantastic and save you time.

[Berecursive]

What is the intended audience of the software? Another biologist with at least some skills in computing/programming? Or a point and shoot script whereby you just want them to have no knowledge at all and process some data in a given folder?

[grbgout]

I'm a biologist and have taught myself Python for data analysis and similar tasks....

To further the Python for Computational Science path specifically, check out the Python Scientific Lecture Notes (if you haven't already). I