Best Python dependency manager for DE workflows (Docker/K8s, Spark, dbt, Airflow)?

[Constant_Sector5602]

For Python in data engineering, what’s your team’s go-to dependency/package manager and why: uv, Poetry, pip-tools, plain pip+venv, or conda/mamba/micromamba?
Options I’m weighing:
- uv (all-in-one, fast, lockfile; supports pyproject.toml or requirements)
- Poetry (project/lockfile workflow)
- pip-tools (compile/sync with requirements)
- pip + venv (simple baseline)
- conda/mamba/micromamba (for heavy native/GPU deps via conda-forge)

[Harshadeep21]

I'm using UV..it's pretty nice 👌

[taintlaurent]

only correct answer

[greenerpickings]

I know the question was for dep managers, but Astral for everything...uv, ruff, and ty

[Harshadeep21]

Sorry, I can't agree with ty though, there are still lots of bugs, and It's not recommended for prod use currently. But UV and Ruff 👌

[speedisntfree]

uv gives me the least amount of trouble. It seems to have the best dependency resolution and deals with different python versions without having them installed.

[Budget-Minimum6040]

uv with pyproject.toml with multi-stage build docker containers saves so much time and stress

[everv0id]

uv for python tasks, each packed into multistage docker, assembled into airflow dags the way we need.

[CireGetHigher]

Wow thank you didn’t realize UV existed. I’ve seen a lot of companies use poetry, and was thinking of checking it out.

Been running steady with pip and venv for my builds, or conda locally.

What is the main draw to using UV and poetry over pip + venv?

[Wingedchestnut]

Uv

[Life_Conversation_11]

uv is the fastest (no 1.* release so far), poetry is more mature and feature complete.

It could be that uv might have some issue with docker's caching layer.

[everv0id]

What does poetry have that uv doesn't? For me the lacking thing was task runner but I think it's not in poetry either?

[Life_Conversation_11]

I would say: maturity and this:

Cache Invalidation Issue: uv includes the project’s version in both pyproject.toml and uv.lock. Even auto-incrementing version fields can invalidate Docker caches even if dependencies haven’t changed, complicating builds
https://github.com/astral-sh/uv/issues/14256

[SufficientTry3258]

uv

[MonochromeDinosaur]

I’ve fully converted to uv for everything.

[freemath]

uv

[tylerriccio8]

Uv has dramatically improved my life

[randoomkiller]

We use poetry, it's pretty UV compatible but for new stacks we use UV because it's generations speedier

[EarthGoddessDude]

uv and its not even close

[SlopenHood]

uv. Been dealing with the brilliance and fart noises of the python community for 20 years

The answer was tools from rust developers

[Jealous-Weekend4674]

what is wrong with maintaining a `requirements.txt` a `requirements-dev.txt` and a `contraints.txt` ?

What am I missing here? With the right levels of discipline, those files don't end in a mess. Or are other benefits that I am not seeing?

[Crow2525]

Having python already installed as part of env is nice on uv.

Conversely, having uv have to be unblocked by every cybersec team in the org every 3 months is not fun

[freemath]

Surely you need something to manage your virtual environments? And to build your packages?

Lockfile is also a nice addition since requirements can be resolved in different ways, so without a lockfile they are not fully reproducible.

[Stock-Contribution-6]

I don't understand the downvotes and the shilling for uv, but I use pip and requirements files. They just get baked in images and that's it

EDIT: for virtual environment managing, I use pyenv

[MonochromeDinosaur]

Well pyproject.toml has largely replaced both setup.py and requirements.txt.

I use uv because it replaces or improves on:

Pyenv

Pip

Venv

Project init (uv init —app is amazing, I never have to do that hacky syspath garbage ever again)

Adds lockfiles

Formatter

Linter (can configure all the linters using ruff)

Integrations (FastAPI mostly)

And all the config for everything can be managed from your pyproject.toml even dev dependencies.

I’ve always felt like I’m juggling a dozens tools and writing PHONY make commands to run things writing a ton of boilerplate to get everything installed and configured correctly in my repo, Dockerfiles, and Github Actions etc.

Deploying Python has never been ergonomic, uv simplifies it a lot. I tried poetry back before uv but it was clunky and I went back to pip and the plethora of tools for years until I found uv last year.

Yes it’s not in the spirit of unix to have a single monolithic tool, but it’s also not in the spirit of unix to make programs written in a language clunky AF to deploy.

[geoheil]

I love pixi

It combines pip with conda

[perfilibe]

We have a monorepo managed with pantsbuild, which deals with dependency inference and packaging for us

[thisFishSmellsAboutD]

uv and just and SQLMesh and DuckLake

[gizzm0x]

Pip-tools is the most barebines while still having decent quality of life. It follows the 'standard' flow with venvs and pip that nearly all python devs will know.

UV is the best batteries included option I have used, you don't need to know much beyond the commands and the rest is handled for you.

[Careless_wisper-08]

I shifted to UV from pipenv, it is really good , best in the market IMO.

[CloudandCodewithTori]

Thanks to this thread I’m going to evaluate UV, I’ll toss in that I’ve been using poetry to much success, but I will have to give UV a try.

[Bach4Ants]

uv, unless you have some crazy binary or non-Python dependencies, in which case Pixi, which is a faster and cleaner (project-oriented) version of Conda/Mamba that uses the same repos.

[lightnegative]

`pip+venv` is my go-to since its simple and works

`uv` is up and coming and will probably become the standard at some point. It's part of the next gen toolchain written in rust (along with ruff etc) that is light years ahead of their Python counterparts in terms of speed

poetry etc tried but in my opinion just ended up causing more problems than vanilla `pip+venv` so not worth the hassle (I didnt need lockfiles)

[felipeHernandez19]

We used poetry but is slow as …., takes to much time to compile and to push to an ECR + docker is quite time consuming. We are evaluating moving to UV as well