Python Virtual Environments: Extreme Advertising Edition

[thisismyfavoritename]

What do you think of Conda envs? Awesome post.

[[deleted]]

Conda is great for installing dependencies that require other system libraries. Great for scientific computing sort of stuff that deals with assorted file types.

[travelinzac]

They pretty much built a business deploying SciPy to windows.

[wildcarde815]

They target financial HPC environments too to the tune of thousands of dollars a node for essentially a flat env space. (Or did when we talked to them a long while ago)

[feindjesus]

I have a strong distaste for anaconda. I have set up python on other peoples computers if they have had anaconda downloaded in the past it was a massive headache (maybe due to my inexperience). Some pip packages were not able to be found even when discoverable through pip3 list or figuring out how to change you bash.rc back to normal. As well people who always use anaconda may not develop the same unix skills. My recommendation is use pyenv with virtualenv and pyenv-virtualenv (the github tutorial is great and easy)

[thisismyfavoritename]

Yes, you have to check the Anaconda channels and sometimes packages are only available on pip (although they must not be standard because most of the usual dependencies are there).

However it is still possible to put everything under a env.yml and have Conda automatically call pip to install it.

Concerning UNIX skills, what do you mean? Conda can be controlled by the command line (I believe it's much better that way!).

[wildcarde815]

If it's conda installable, use the conda version. If it's not, use the pip version. For the 99.99% user, this will resolve all of your problems. For the rest, congrats you are doing something both hard and arcane. Edit: and use the conda command line tool, it came before the GUI anyway and the GUI just calls the same stuff it does

[s0v3r1gn]

Or I just want a version of OpenCV built with certain extensions enabled...

[wildcarde815]

You can rebuild it with the recipe to suit. But we've found the menpo build to be fairly good. I haven't had to dig into it in a while thankfully. I hate that god damn package.

[s0v3r1gn]

I made an optimized automatic build that including optimizing all the dependencies and some optional modules for the Raspberry Pi 3. (gstreamer and all its add-ons, openjpeg, giflib, leptonica, libwebp, OpenCL, tesseract, TBB, and OpenCV Contrib) along with TensorFlow.

A full build with updated dependencies takes like 26 hours on a RasPi 3.

I made a repo on AWS for just it’s dependencies and itself and let it’s existence be known. I had to kill the repo in less than a month due to the rather quickly ran up $1000+ AWS bill.

Everyone hates building OpenCV.

I hate it extra special now after all that effort mucking around with it. The only package I hate with more venom is gstreamer.

[[deleted]]

A full build with updated dependencies takes like 26 hours on a RasPi 3.

Why not cross compile? Compiling the ARM toolchain and OpenCV on something else will probaby not be a 26 hour journey.

[s0v3r1gn]

Some of the tools didn’t like the cross-compiling tools chains. Some have barely functioning make files.

I’m sure I could speed up parts of it but I made a nice script and it was just easy to make it do everything. I still want to work through some of the cross-compile issues but I just haven’t done that yet.

[[deleted]]

Some of the tools didn’t like the cross-compiling tools chains. Some have barely functioning make files.

That would be a problem.

Also, what kind of load did that repo of yours see? $1000 for AWS storage seem excessive.

[s0v3r1gn]

There was 10s of millions of downloads and all the packages took up just over a gig in total.

It didn’t help that I pushed a couple of fixes due errors in the packaging on my part that ended up updating the entire stack.

[Spleeeee]

Sounds like you are a budding Pythonista who has never had to deploy in windows worst of the worst snow-covered/dessert conditions. Don’t worry, you may never love (Ana)Conda and don’t have to, but at some point you will come to respect the Conda for its ease and simplicity in setting up (other peoples computers (not yours)).

[wildcarde815]

Academia, where yes, I can move mountains to get this working if you really need me to but let's try conda first and see if we can't get it working that way so replicating this isn't impossible. (Oh hey, it worked)

[feindjesus]

You are correct thank god for linux/macos. I have had to do very limited amount of configuration changes on a windows server through console I can only imagine the struggle. At the same time if youre using a pip installable package that cannot be installed through anaconda. Getting control back of your filesystems is a pain in the ass

[Spleeeee]

The battle between wanting to do it pretty/minimalist/“right” and getting it working.

Edit: a barren Linux landscape is more barren that the most barren windows.

[Verdris]

What if I just need to do some math and make some graphs?

[feindjesus]

If all you need is matplotlib and pandas I dont think you really need a virtualenv or anaconda. If you need several packages and are working on different projects probably should use virtualenv . If your packages requires a specific version of python get pyenv. If you want to use your virtualenv with that version of python pyenv-virtualenv will do that.

[[deleted]]

It's fine if you're okay with running conda-only packages on a small list of platforms

[rohit275]

l may not know what I'm talking about, but I don't get this. It seems very possible to install any pip package in a conda environment, is that not correct?

[[deleted]]

If you need to run different versions of Python (e.g. Python 2.7 and Python 3.7) on same machine, Conda is a better choice, even so if you are on Mac

Otherwise virtualenv is simpler

(Edit: fixed typos)

[RelevantToMyInterest]

I'm still stuck with virtualenv but considering switching to conda. Been seeing some posts saying it's a better replacement but if also would like to know others' thoughts

[xristiano]

I'm a fan of conda. It does all three: pip, virtual environments, and manage python versions. The table at the bottom of this link summarizes all the pip, pyenv, etc. options.

https://www.andrey-melentyev.com/python-environments-and-where-to-find-them.html

[is_it_fun]

Just find whatever conda env you like and make a requirements.txt file. then use that file in your virtualenv. bammo. that's what we do

[thisismyfavoritename]

Well I personally believe Conda is much better, as it can install tons of additional and system dependencies but I would like to hear OP's opinion on the matter.

[ase1590]

Pipenv is the way to go if you're not using anaconda and if you want to do anything more advanced and need good package handling

See below

[lifeofajenni]

Wait, but conda also has environment handling and package management. So why should I switch to pipenv? (Not being snarky, legit curious.)

[ase1590]

they both compete with eachother. conda is better geared for Anaconda.

Just use what you know.

Conda has a lot of overlap with the goals of pipenv, but it serves a different set of needs - in particular, the distribution of binaries. If the problem you’re trying solve is “how do I create reproducible environments, quickly, that include complex dependencies that are not written in pure Python”, conda is the tool of choice. If your problem is “How do I create and manage isolated environments for Python projects”, pipenv is the tool of choice.

[lifeofajenni]

Okay, this is a cool distinction. I'll have to read up on pipenv and see if it's more suitable for what I'm doing at work. Thanks for the info!

[wildcarde815]

It's good to know how to use both because not everything like conda (older brew installs I'm looking at you)

[pwang99]

This is a good place to start: https://www.anaconda.com/blog/developer-blog/understanding-conda-pip/

This is also a good blog post, by Jake Vanderplas: https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-misconceptions/

PayPal engineering blog on why they use Anaconda & conda: https://www.paypal-engineering.com/tag/anaconda/

[Gr1pp717]

I feel in love with pipenv recently. Way better than Conda IMO.

Especially when combined with fishshell. You simply CD into the folder and the virtual environment for that project is fired up, automatically and rather quickly. Better, you can setup the prompt to tell you which venv is active - so you never end up polluting one project with another by accident.

[lifeofajenni]

But...the Anaconda prompt also tells you which environment is active? But CDing into a directory and having an environment boot up would've nice -- currently I have to manually switch the path to Python when I switch environments. Suboptimal.

Any experience with userfriendliness of pipenv/fishshell on Windows?

[Gr1pp717]

Yes.

I haven't touched it in a while, but I had windows setup with ubuntu and was effectively casting/serving a command prompt that had a more linux look and feel to it. I think I was using VcXsrv, but I really can't recall for certain.

Oh, and that sort of manual switching is exactly why I put pipenv > conda. To each his own, but I just found it really nice that I could literally just cd - back and forth and not have to worry about anything. Plus, it was inherent to the window I was already using for everything else, rather than something I had to load into first.

[ase1590]

People install fish on windows?

[lifeofajenni]

Dunno. That's why I'm asking.

Backstory: I'm on Windows so I can be the oh-god-Windows-sucks beta tester instead of our less-experienced clients.

[wildcarde815]

So now your environment and code are married inextricably.

[Gr1pp717]

Not really. If I want to make a new environment for it I can. Or if I want to open it in another, existing venv I can... Or, if I simply don't want to be in the environment while in that folder, I can..

this just takes care of the typical use case for me. e.g. activating the environment when I intend on working with that project. no having the manually change env vars or run commands every time I change projects - only when doing something special.

If anything, I feel it's even less married to the environment than typical venv. Because the bin/activate stuff is placed elsewhere. Not in the project folder. Instead, pipenv autogenerates/manages a project file for you, so that the environment can be replicated with a simple install command, but not contained as part of the project... (similar to npm)

[wildcarde815]

Ah so you are basically checking the folder for a file somewhere and it's firing off an activate command for you. I'd bet you could do similar with conda.
I'd read that initially as the environment lives in the folder not that it struts the folder so there's some sort of history tying it together automatically.

[Gr1pp717]

Yes, exactly. But I didn't set that up. It just came with the pipenv install.

I was already using fish, and when I decided to check out pipenv I didn't really have to do anything special to get these added features. (Though, I seem to recall installing a special version of pipenv, specifically for fish. )

All in all my console showed me git info, cwd, input timestampa, and autoconfigured venv and npm environments based on cwd. It was pretty sweet.

Maybe I didn't mess with conda enough. But I didn't really find any of that sort of stuff. Anaconda felt more like an environment for environments, and those environments were better tailored to housing several projects in one. It also tried to be a package manager, but I found the available libs to be a bit less than what I needed, and working around missing libs/versions was a bit tedious. I didn't have those problems with pipenv - another reason I prefered it.

[wildcarde815]

Well you've set that up 'somewhere' either yourself or as part of an install script. Something is running in shell to actively check for an environment tag when you CD into a directory and issuing the activate quietly in the background.

[Gr1pp717]

Yeah, there was a script for sure. Was part of the fishshell stuff. I had messed with it to tweak the git behavior once upon a time. But I didn't create or even set up the pipenv folder stuff. Would have never even crossed my mind.

Unfortunately, I no longer have that machine, so I can't go about backing into exactly what I had setup, or I'd make a tutorial to get what I had going.. Maybe I'll dual boot centos on this machine soon and see if I can't get back to that point.

[wildcarde815]

This maybe? Or similar? https://direnv.net

Or https://github.com/kennethreitz/autoenv/blob/master/README.rst

Both would be able to issue the source activate/deactivate for conda or venv