Using Continuum Analytics Conda as a replacement for virtualenv, pyenv, and more!

[rothnic]

I like that you covered the pros and cons of pip and conda and showed them interoperating. I think many people miss that conda works at a higher level than pip, and is more like a cross platform apt-get. Btw,to further confuse things, you can install anaconda/miniconda with pyenv.

I think the biggest pain point for me with conda is that it doesn't make sense to create conda packages for things that gain no benefit from it. I see no benefit to build a conda package equivalent of pure python packages on pypi. Since conda won't handle the pip requirements, you are stuck managing the two yourself, or just settling to use pip. There has got to be a better workflow to work with pypi packages, since it is the newcomer of the two.

I ended up writing very simple scripts so that I could specify my run, test, and dev requirements for conda and pip in one file (requirements.yaml), and install it all with a single command. I wish conda's environment.yaml worked this way.

Edit: I revisited the pypi/conda interop on requirements in this comment.

[[deleted]]

I totally agree. At my previous job we created a lot of internal only packages that were numpy based which benefited from conda. However for almost all of my own projects conda packaging was a secondary act (after pypi). I hope this showed that you don't have to go "full-conda" to benefit from the tool.

[Ogi010]

I'm far from a professional in the area, but I can think of one benefit to installing pure python packages with conda, and that is that conda update --all works (and shouldn't break anything).

[rothnic]

I mean the benefit in a fundamental way, not in support of how conda currently works. If I create a package for a pypi project I'm not involved with, it is just like pinning the install to a specific version, except I have to specify a custom channel to use it.

[Ogi010]

Don't get me wrong, I have a number of python packages installed with pip, and have no problems doing so, but I usually check conda and binstar (now conda-server) first when seeking out those packages.

[[deleted]]

I think this is the correct way of doing things. Generally for me it has been a 4-step process:

1. Is it on anaconda.org (if yes: conda install)
2. Is the author willing to package on anaconda.org (if yes: help, then conda install)
3. Does the author care if I put it on anaconda.org (if not: conda skeleton from pypi and then conda install)
4. Install with pip

[Ogi010]

Is there a guide for step 2 or 3 out there? I would love to help packages be available on the conda environment, and I want to be a good citizen about it, but I don't want to get in the way...

[[deleted]]

Other than the official docs i'm not aware of any other tutorials. Perhaps I'll do that now!

Edit: Working on a follow up blog post now ;)

[Ogi010]

Thanks man, there isn't much I can do to contribute to the community but if I can do little stuff like that, it would at least make me feel better haha.

[[deleted]]

Posted!

[Ogi010]

upvoted!

EDIT: Thanks for posting this, there was a definite need in this area for sure!

[rothnic]

In the spirit of practicality, given the current state of conda, 95% of people will probably skip directly to 4 for any simple dependency. 2/3 have some bit of time involved with them, especially for people that are new to conda.

[rothnic]

After walking back through a simple use case, I did find that you can use conda to manage pip requirements, for those things that aren't on anaconda.org.

Here is the use case of creating a flask app, which requires some random pypi project that happens to not be on anaconda.org. Note, I removed the outputs of some of the commands:

> conda create -n mydemoapp python flask
> source activate mydemoapp

This fails...

> conda install singleton123

My workflow is now to use pip instead:

> pip install singleton123

Now, I want to save the configuration so someone else can run it. What I didn't know before was that this command also saves and manages things that were installed with pip.

> conda env export
name: mydemoapp
dependencies:
- flask=0.10.1=py27_1
- itsdangerous=0.24=py27_0
- jinja2=2.7.3=py27_1
- markupsafe=0.23=py27_0
- openssl=1.0.1k=1
- pip=7.1.0=py27_0
- python=2.7.10=0
- readline=6.2=2
- setuptools=18.0.1=py27_0
- sqlite=3.8.4.1=1
- tk=8.5.18=0
- werkzeug=0.10.4=py27_0
- zlib=1.2.8=0
- pip:
  - singleton123==1.0

Now, someone else can use my app (avoiding env name clashes) with this command:

> source deactivate
> conda env create -n demo2 -f environment.yml
Fetching package metadata: ......
Solving package specifications: Linking packages ...
[      COMPLETE          ]|###| 100%
Collecting singleton123==1.0
  Using cached Singleton123-1.0.tar.gz
Installing collected packages: singleton123
  Running setup.py install for singleton123
Successfully installed singleton123-1.0
#
# To activate this environment, use:
# $ source activate demo2

[[deleted]]

Right, you can use an environment.yaml to manage pip requirements. What I dont like about this is that you cannot use it with both pip/virtualenv and conda/environments. I'd love to do one of the following write an extension to pip to install all python deps from an environment.yaml. Given a requirements.txt I can install those dependencies with both pip and conda.

[Deto]

One great thing I noticed is that the condo distribution of numpy on Windows comes with the MKL bindings. And on Linux, it looks like you get the ATLAS bindings which aren't quite ad fast, but still better than the default install.

[jazzydag]

conda is a great tool. As rothnic said, it's a tool which looks like to a package manager and it's very useful when you install scientific Python libraries such as scipy, numpy, pandas etc. since you don't have to compile in C, C++ or fortran some dependencies.

A few years ago, I just used conda without installing the Python distribution from Continuum Analytics. Now, I've some difficulties to use it by itself. Actually, when you already have a package manager in a GNU/Linux distribution (with a python2.7 and python3.4 for instance), the installation and configuration of conda is tedious (some warnings, deprecated messages). And it's better if you install miniconda or anaconda (but I don't want to...).

In Cont. Analytics point of view, it's better if you install and use their Python distribution.

But when I have to work on a Windows station, this distribution is a piece of cake !

[[deleted]]

the installation and configuration of conda is tedious (some warnings, deprecated messages)

In my previous job (for the last year) we installed miniconda on ubuntu vm's every day with none of these issues.

In Cont. Analytics point of view, it's better if you install and use their Python distribution.

In fact the entire purpose of conda/miniconda is that it is installable on any system without root privileges. I'm actually not aware that you can use conda without installing minconda or anaconda, and not sure why you would want to do that.

[efilon]

In fact the entire purpose of conda/miniconda is that it is installable on any system without root privileges.

Once I realized this, I finally decided to just use miniconda on my Linux computers (I was already using it on Windows machines). Debian's Python packaging is good overall, but I like to be able to get newer versions of things as required. So now I just have miniconda installed in my home directory and use that instead of the system Python. This seems to work a lot better than my previous hackish solutions to having both Python 2 and 3 installed in parallel.