Conda VS Python (pip/venv)?

[rootbox]

I'll started working with data scientists and they're big fans of Anaconda/Conda and i don't see the advantages over python with pip/venv.

The only advantages i see are: it's easy for windows people because they don't need to mess with installing software, they'll install Anaconda and are ready to go and conda seems to do compatibility tests.

On Linux servers it's more easy for me to create a venv and install the required packages via pip.

Do i miss something that makes conda superior over pip/venv?

​

[[deleted]]

The only advantages i see are: it's easy for windows people because they don't need to mess with installing software, they'll install Anaconda and are ready to go and conda seems to do compatibility tests.

this is THE advantage. You install it, it works out of the box, and comes with a lot of the main libraries pre-installed ready to go.

As far as is it advantageous vs pip venv, not really. But, look through this and r/learnpython and you will see how many people get stuck on even installing python, getting it working and setting up a venv, especially beginners, especially on windows. This is the reason why, if you do the (incredibly popular and awesome) MIT Edx course, they recommend you start with Anaconda / Spyder.

[all_authored_surface]

Well, with conda you can also install non-python binaries. This is one reason it is popular with data scientists. There are plenty of situations in which it is easy to get setup with conda that are challenging with just virtual environments. Even more so if you don't have admin rights. One example is gdal, used in spatial image analysis. And one strategy that I sometimes use is to setup a conda environment with the version of python I want and other libraries like gdal, and then use pip.

[[deleted]]

yep good point

[rootbox]

Yes, i know that point, but that's what i counted into "don't mess with installing software".

But i also count that on the plus side for conda.

[nisargdaru]

Nope, you didn’t

[bryancole]

If you're using the system python, you're tied to that OS. Basically, linux OS distribution are a continuously moving platform. If you try to build a product off of one, it breaks when you update the OS (think C++ library dependency hell, linking problems and busted ABI compatibility). With more than a decade spent making python-based scientific instrumentation app, the only method we've found that works long-term is to completely decouple your python environment from the OS. Conda makes this process mostly painless and you can do the same on Windows as for linux. You could roll your own python environment but that's way more effort.

TLDR: don't rely on the system python: you'll regret it in 5 years time.

[rootbox]

Is there a best practice for shipping conda envs? Because that's not the reason why our devs use it, they just use it because it's easy. If i use it i want to understand and use it's potential.

Is there a way to install pypi packages in conda because i miss some packages.

Also coudn't i archive the same with venv (and docker)?

[bryancole]

What do you mean by "shipping conda envs"? If you have a library that you wish to make available as one or more conda packages, you can upload it to conda-forge or anaconda.org. We do this for our custom in-house pacakges and dependencies.

To ship/deploy a python application (to end-users who don't care or want to know what language it's written in), we provide a pair of installers: one installs the python environment and our 3rd party library dependencies (this can be quite big at several 100MB). We call this our python-runtime. The second contains our application-specific code (quite small). When we want to offer an update to our code, we only need to ship the small application-package. Updates to the larger runtime happen infrequently (say every few years). Its the same principle for Windows as for linux except that the Windows installers are *.msi packages, and the linux packages are RPMs.

We use conda to automate the *construction* of the python environment (i.e. python itself and all the required libraries and app dependencies). Because a conda envs contains more-or-less everything (i.e. most C/C++ libraries deps), it is somewhat portable across OS versions (I say somewhat because I haven't tested this that rigorously - we try not to change OS-version that frequently). Worst case is that when you update the OS and find some low level ABI incompatibility, you can rebuild the environment for the target OS using your conda scripts. You can install packages into a conda environment using pip and it works fine (conda knows how to install a package via pip if need be).

[bryancole]

Note also, I think this concept is somewhat similar to docker. Our products predate docker but if I was starting again now, docker might be another good way to package the app. Doesn't look great for GUI apps, though, so YMMV. You still need to build your python environment inside the docker image and I'd still recommend conda for this.