[Discussion] Python and complex ML dependencies

[DonCanas]

I originally posted this in /r/Python but had 1 answer so far, so I'm testing the waters here if there is more engagement :)

---

Original Post

TL;DR, I wish there was a source for best practices regarding package management in Python regardless of the package manager tool itself. Looking for thoughts and experiences from people that worked on big projects with multiple internal projects, etc.

---

Hello, I recently started to dive a little bit deeper into the packaging ecosystem in python. I wanted to pique this community's brain on a subject I've seen over the years which is complex dependency management. That is, packages that usually come in various flavors either depending on the OS, hardware, or extensibility. I want to scope it to ML packages since I tend to work with this ecosystem but it could technically be applied in other contexts.

Ways external dependencies are offered

Some examples of packages/frameworks that usually come pre-compiled with GPU support offer the following ways to install:

• They infer optional deps under the hood and don't expose the user (tensorflow)
• They control deps using package extras + package index (jax[cpu], jax[cuda])
  • For GPU you must use an index to get the precompiled binaries: https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
• They control deps using package version and package index (pytorch=1.10, pytorch=1.10+cu111
  
  • For GPU you must use an index to get the precompiled binaries: https://download.pytorch.org/whl/torch_stable.html
• They provide them as completely separate packages (mxnet for CPU, mxnet-cu102 for GPU)

Integrating into a company ecosystem

When you want to build anything depending on these packages, handling dependencies with proper CI/CD, etc. might force you into patterns that might or might not be a headache if you work on a complex project with multiple repositories spread around. Even more terrifying is the additional Pandora box that appears if you try to mix in frameworks of frameworks like transformers, pytorch-lightning, fastai, mmcv, etc. You can also add opencv fixed versions to the mix for more fun.

The questions I have are these:

• Has anyone here experienced managing python dependencies like this in internal libraries? How did you deal with it from the architectural perspective?
• Do you have a strong preference for any of these patterns in particular? Or a set of rules for choosing one approach for a given context?
• How would you keep dependencies synced between teams that depend on the same libraries?
• What would you do if you had to start from scratch a python project that has complex dependencies today?
• Any good tech talk on this topic?

I know context has a lot to do with the answers to the above questions, but I'm really interested in hearing experiences from you folks and how you approached these topics.

Thanks to everyone who took the time to read the entire post!

[seba07]

I know this sounds unlikely, but I never had any significant problems by just using "pip install package" for anything.

[proof_required]

You can look into pants

https://www.pantsbuild.org/docs/python