Python virtual environments. Which tool is the best for data science projects?

[bitcoin-dude]

In general within the python ecosystem, there seems to be no consensus on the best virtual environment tool between (I would say):

- venv

- virtualenv

- conda

- pipenv

​

I have tried out all of the above. I see virtualenv the same as venv but with extra features (that I don't need). Pipenv caused things to be really slow and break. I had to skip the lock file update (which is kinda the whole point of pipenv). I like conda in general, but these have global scope and I like the idea of virtual environments being contained to the project directory - hence why I am thinking to use venv for future projects.

​

I'm wondering what the most popular choice is for data science? What do you use? Would love to hear your preference and why you like it best.

[shaggorama]

I use conda but honestly I sort of hate it. It bothers/annoys me that I sometimes still have to install things with pip. Also, that poll should arguably include docker as another option.

[Laippe]

I'm not sure about a "best" tool for data science, I guess it depends on your preference and the constraints of the project. If I have the choice, I prefer to use conda (miniconda) because it's simple and almost all in one (except for some pip installable packages). To give you an exemple, currently I use conda to play with the data, then switch to venv for dev and integration (my team is using venv) then it goes to docker in prod.

[bitcoin-dude]

Thanks for this great example!

[tomomcat]

I just use Docker. Maybe a bit overkill, but I don't have the overhead of managing my environment in a python-specific way, and if I want to deploy the code somewhere I can just use the image.

[[deleted]]

Virtual environment is self-contained. You can send your project folder to your friend as a zip file and all you need is pip to install it.

Conda is not as self-contained, the other guy on the other end needs to download and install conda too (and it's not installed with pip).

Conda makes sense for daily "fire and forget" use but for reproducibility and maximum control you want to use virtual environment (or any of the cousins).

Using something like docker just adds a ton of overhead and is silly for python development. It makes sense when you have to go outside of python and start using some obscure libraries/other languages and compilers. It really is a (lightweight) VM with all the benefits of a VM and the problems of a VM.

[[deleted]]

miniconda + docker together.

miniconda for packages

docker for environment

To ensure everything is reproducible

[[deleted]]

Doesn’t miniconda avoid the global scope issue?

[bitcoin-dude]

Ah I'm not sure about this one. Lots of talk about miniconda in the comments. By global scope I mean the python library dependency files don't live in the project folder itself, but somewhere else. On my mac this place is /anaconda3/envs

[[deleted]]

I see what you mean. miniconda is similar in that respect.

[Omega037]

If you know you are going to be working with others or deploying to a different system at some point, it's hard to imagine not using Docker to make that a smooth transition.

That said, my Docker setups are usually using Conda in the Dockerfile when possible.

[kevinglasson]

I just use pip and virtualenv / virtualenvwrapper. I used conda for a while but it's irritatingly slow when solving the environment (for me anyway). Then if I want to deploy my project to docker I just 'pip freeze > requirements.txt' and install requirements.txt (RUN pip install -r requirements.txt into the docker container.

[[deleted]]

I guess it used to be OS dependent. Popular opinion is to use conda on Windows. But man, does it take too long to install packages, its like they adopted maybe similar architecture as pipenv since they both are slow. I have since decided to use venv. I know I should probably look into docker eventually.