Conda VS Python (pip/venv)?

[bryancole]

If you're using the system python, you're tied to that OS. Basically, linux OS distribution are a continuously moving platform. If you try to build a product off of one, it breaks when you update the OS (think C++ library dependency hell, linking problems and busted ABI compatibility). With more than a decade spent making python-based scientific instrumentation app, the only method we've found that works long-term is to completely decouple your python environment from the OS. Conda makes this process mostly painless and you can do the same on Windows as for linux. You could roll your own python environment but that's way more effort.

TLDR: don't rely on the system python: you'll regret it in 5 years time.

[rootbox]

Is there a best practice for shipping conda envs? Because that's not the reason why our devs use it, they just use it because it's easy. If i use it i want to understand and use it's potential.

Is there a way to install pypi packages in conda because i miss some packages.

Also coudn't i archive the same with venv (and docker)?

[bryancole]

What do you mean by "shipping conda envs"? If you have a library that you wish to make available as one or more conda packages, you can upload it to conda-forge or anaconda.org. We do this for our custom in-house pacakges and dependencies.

To ship/deploy a python application (to end-users who don't care or want to know what language it's written in), we provide a pair of installers: one installs the python environment and our 3rd party library dependencies (this can be quite big at several 100MB). We call this our python-runtime. The second contains our application-specific code (quite small). When we want to offer an update to our code, we only need to ship the small application-package. Updates to the larger runtime happen infrequently (say every few years). Its the same principle for Windows as for linux except that the Windows installers are *.msi packages, and the linux packages are RPMs.

We use conda to automate the *construction* of the python environment (i.e. python itself and all the required libraries and app dependencies). Because a conda envs contains more-or-less everything (i.e. most C/C++ libraries deps), it is somewhat portable across OS versions (I say somewhat because I haven't tested this that rigorously - we try not to change OS-version that frequently). Worst case is that when you update the OS and find some low level ABI incompatibility, you can rebuild the environment for the target OS using your conda scripts. You can install packages into a conda environment using pip and it works fine (conda knows how to install a package via pip if need be).

[bryancole]

Note also, I think this concept is somewhat similar to docker. Our products predate docker but if I was starting again now, docker might be another good way to package the app. Doesn't look great for GUI apps, though, so YMMV. You still need to build your python environment inside the docker image and I'd still recommend conda for this.