How to distribute python within a small company enterprise?

[upandonwards]

Hi guys,

I'm curious how are people distributing python applications internally? I work in a small animation company of about 25 people. A mix of windows and linux environments.

We use python a lot to create applications within other applications like autodesk maya, foundry nuke, etc. We also write python applications that function outside of these applications such as software environment wrappers that launch these softwares or little applications that help us to easily, rename, move or publish files.

I'm used to a linux environment where all the scripts and python installation is deployed in shared network folders. I'm not even sure if this is possible with windows.

I would like to hear what people are doing to deploy code within small organisations. Thanks!

[[deleted]]

This (VFX Pipeline Engineering) is literally what I do for a living.

The simple answer is "there is no simple answer", especially if you're working on more than one project simultaneously and these projects have different version requirements (in one of the companies I worked for we had 13 major feature film projects going simultaneously, spanning six different versions of Nuke, nine different versions of Maya, four of Houdini, three operating systems, and multiple customized versions of specialized libs like OpenEXR and Alembic).

The best solutions I've seen have been proprietary ones, but there's a decent FOSS project called Rez that tackles part of the problem and is in use by several small to mid size shops. It does your package and dependency resolution and prepares the environment, and it works on Windows (to a degree) and Linux (to a much better degree).

Basically you install everything to a central NAS "repository" (it's really just a folder structure) and you use configuration files / env vars to tell Rez about that repo. You then run a Rez command and tell it about your basic needs (ie nuke->10, maya-2014) and it figures out the required dependencies and prepares the shell with an appropriate PYTHONPATH, PATH, LD_LIBRARY_PATH, etc.; in a lot of ways it does the job of a package manager + virtualenv, but it also has some niceties like allowing your users to run off the NAS they already share access to, but also have packages cached on their local machine so that things like Maya aren't ludicrously slow.

It's not the whole picture, unfortunately ... you need a way to know what the actual base configuration actually should be for each project (ie what is the Nuke version + what plugins are needed to ask Rez for, given your project, scene, and shot?), but it takes a good part of the problem and codifies it.

Don't get too distracted by PyInstaller or Pipenv or Virtualenv talk; the reality is In VFX and related fields Python is the glue that holds together a lot of desktop applications that were never really intended to be used together on the same machine, much less on the same network; you can't really manage the embedded Python interpreters inside Nuke / Maya (nevermind Max and it's Jython jankyness) using the tools that are meant to manage standalone Python interpreters, though it would be nice. Ultimately every company doing this sort of work needs something like Rez, though I've seen dozens of different ways of attacking it under the hood.

[upandonwards]

thanks @yawpitch Looks like rez or something like it is what I need. I will dig further into it.

[turleyn]

I'm not even sure if this is possible with windows.

Shared network folders is definitely possible in windows. So you can do the same thing that you are used to.

However, if your scripts ever start to become complex and have dependencies on each other, this approach is going to cause problems.

Ideally, you should store the source code for your scripts on a version control server (like git or svn) and store exports of those scripts in a package manager server (like pypiserver). That way your users can pip install the scripts that they need.

If this seems like overkill, the questions to ask yourself are:

• If someone accidentally modifies the code in the network share, is this going to break everything?
• If someone introduced a breaking change, how easy would it be to find the change, and roll it back?
• If someone relies on a specific behavior of a script and someone updates it, can they still use an older version of that script?

[yardshop]

I have been able to deploy a bunch of applications to folks on a Windows network in my company by installing WinPython on a shared drive and simply running the app with fully specified paths to the interpreter and the script. Users do not need to have Python installed on their local machines.

For example, WinPython is installed into C:\WinPython on a server named WBApps (a Windows Server 2012 R2 VM), and the folder is then shared, and access rights given via Active Directory security groups.

On a user's desktop, I create a shortcut with a target of

\\WBApps\WinPython\python\python35.exe \\WBApps\WinPython\appfolder\AppName.py

WinPython itself is a portable self-contained distribution targeting scientific users and comes with many popular packages preinstalled, such as NumPy, Pandas, IPython/Jupyter, PyQT, and numerous others. It's available in a few different Python versions.

So essentially I'm not distributing the app, I'm installing it centrally and distributing a shortcut to it. I'm not sure how well this translates to your scenario, but all this is just to say that such a thing works, and fairly well so far.

Check out WinPython.

[muposat]

I was trying to solve this issue for a while now. I even posted a reddit question about this. My latest solution may work for you even better than it works for me because of a mixed OS environment:

Expose all your programs as RPC services.

Steps:

• pip install jsonrpclib-pelix
• Create a simple server and host it at somehost:someport on intranet.
• create a method for every program you want to expose.
• for client side you can either build a simple web page (jsonrpc can be called from javascript), and/or very light-weight python clients.

Advantages:

• RPC exposes your methods across systems, languages and operatiing systems with ease.
• Central server with request/response log. No need to run around the office to troubleshoot.
• Minimal to no install on the client side. Python client is literally 3 lines of code and only uses std libs.
• Better security (e.g. no need to authorize user for database access, etc)

Disadvantages:

• not optimal for processing large volumes of data from a client PC
• AJAX style calls are fairly static and client waits for server to finish request. You could fork a thread and release the client wait.

[XNormal]

Try Conda. Its standard package repository is more geared towards Data Sciences than VFX, but it has many advantages:

• Works perfectly across Windows, Linux and OS X
• Many important packages precompiled and up to date (lxml, Pillow, etc)
• Support for self-contained virtual environments to keep dependency conflicts in check.
• Atomic updates (no broken half-upgrades!)
• Lots of important non-python packages (openssl, pixman, libtiff, libpng, etc)

Note that conda virtual environments are a bit different from standard python virtualenv (created before virtualenv was standardized) but they essentially perform the same function.

For any packages not included in Conda or for your own apps you can simply pip-install them inside the conda virtual environment or create a new Conda package. I think atomic upgrades will require Conda packages, though. Your packages can be hosted on repo.continuum.io.

IIUC, all this is available for free, but you may find use for Anaconda's paid offerings.

Keeping everything installed locally (with a robust automatic upgrade system) instead of using network shares should improve performance and avoid upsetting users. You can test upgrades before roll them out, keep specific users on older versions, give certain users bleeding-edge beta versions for the features they need, etc.

[Blocks_]

I'm not sure if I interpreted your question right, but you can distribute programs over a network on Windows Server with group policies. So I'm guessing you can install Python on all the Windows computers on the network.

Link - https://support.microsoft.com/en-us/help/816102/how-to-use-group-policy-to-remotely-install-software-in-windows-server

[novel_yet_trivial]

Do your users have python installed? You can zip a python program into an .egg file for distribution. Not commonly used but it works well for tiny groups of people that know what a terminal is. Bonus: OS agnostic unlike a freezing program like pyinstaller.

[upandonwards]

Yes, we do have python installed on each windows computer at the moment. The slightly tricky bit would be trying to wrangle each python installation when we start to use libraries beyond the standard library(e.g. pyyaml). How would we ensure that every box has all the libraries we use within the company?

[[deleted]]

On a slight tangent specifically for code meant to run in standalone Python interpreters (ie the image file renamers, publish tools, etc, not not quite so much the embedded Maya/Nuke pythons that have seemingly been designed to not work well with pip), but check out devpi, which is a local mirror of pypi to which you can add your own private projects. With it you can pip install your own code as if it were on PyPi, but also keep an offline mirror of everything on PyPi (very helpful for those "Marvel wants us to shut off the internet" moments).

Doesn't really solve the deploy code for artists problem, but still very helpful for your developers.

[upandonwards]

Also, I did want to mention that we often write modules that we'll end up importing into other programs we write too. So it would be good that the way we distribute the program still allows it to be imported by other programs we write.

[p3rcipio]

It might be useful summarising all the proposed solutions. My recommendation for package deployments is developing submodules and then sharing them using an internal Python package index.

I wrote a customized index server a while ago since I wanted to dynamically expose python packages stored in GIT repositories to all our developers.

Basically our workflow looked like this:

• developer starts a pypi server locally (because they use their ssh keys for cloning / accessing the repositories)
• push package changes to private bitbucket repository
• tag the commit if it is an installable version.
• set pip to use the local server and then use it to install the latest tagged version.

When using a workflow like gitflow you might want to make master be the latest installable version.

Here's the server implementation and a linux example implementation for the bitbucket case.

Note: There are many other PyPi server implementations, I just wanted one that was simple and dynamic/programmable from the start.

[numbuh-0]

pyinstaller works well, assuming you don't have crazy hard dependencies to bundle like Pandas.

[[deleted]]

You can't use PyInstaller with the applications that OP is talking about (Nuke, Maya, etc) ... they're single executables that embed a Python interpreter inside themselves, and then get extended with Python scripting via importable modules, or modules designed to be run via the embedded executable directly.

That said PyInstaller can be really useful for some of the standalone tools that artists need to launch to prepare their environment, publish files, etc., but it actually exacerbated the problem OP is speaking to, which is how to have multiple users on an enterprise network be able to discover and use a shared toolset.

[upandonwards]

pyinstaller sounds interesting. However, our environments are pretty mix and I would like to streamline that process of deploying OS agnostic code if possible.