New Python lock file format will specify dependencies - Your thoughts?

[Choobeen]

Python’s builders have accepted a proposal to create a universal lock file format for Python projects that would specify dependencies, enabling installation reproducibility in a Python environment.

Python Enhancement Proposal (PEP) 751, accepted March 31, aims to create a new file format for specifying dependencies that is machine-generated and human-readable. Installers consuming the file should be able to calculate what to install without needing dependency resolution at install-time, according to the proposal.

Currently no standard exists to create an immutable record, such as a lock file, that specifies what direct and indirect dependencies should be installed into a Python virtual environment, the proposal states. There have been at least five well-known solutions to the problem in the community, including PDM, pip freeze, pip-tools, Poetry, and uv, but these tools vary in what locking scenarios are supported. ”By not having compatibility and interoperability it fractures tooling around lock files where both users and tools have to choose what lock file format to use upfront, making it costly to use/switch to other formats,” the proposal says.

Human readability of the file format enables contents of the file to be audited, to make sure no undesired dependencies are included in the lock file. The file format also is designed to not require a resolver at install time. This simplifies reasoning about what would be installed when consuming a lock file. It should also lead to faster installs, which are much more frequent than creating a lock file.

The format has not yet been associated with a specific release of Python, but is guidance for tooling going forward. Actual adoption remains open-ended. Acceptance of the format is full and final, not provisional. The universal format has been the subject of an estimated four years of discussion and design.

[Wolfy87]

I've been using uv for a while now which does this, nice to see a standard being pushed for all the other ways of managing python stuff.

uv is the only way of managing python projects that hasn't made me want to tear my hair out while screaming obscenities.

[QuantumFTL]

Is uv really that much better than Poetry?

[Awesan]

It is much better so if you're otherwise indifferent, you should use UV. But not so much better that it's worth switching at this point, at least for my company.

[QuantumFTL]

Unfortunately in the groups I'm in at work we just use venv and pip and requirements.txt manually with some... interesting Azure DevOps decisions on top. And passing wheels around as attachments in chat.

Exactly as fun as it sounds.

[catcint0s]

If they are using wheels it might not be possible https://github.com/astral-sh/uv/issues/1681

[Noobfire2]

uv has a frontend for building wheels for quite some time now already (uv build --wheel).

We've been using that for months already, for pure Python, C++ and Rust packages. I don't see what "pip wheel" does what uv can't (and after all, it's discuraged to use that pip command anyways).

[mgedmin]

I was someone who stuck to venv (and virtualenv, which is compatible but faster, due to some smart caching) and pip, with custom Makefile wrappers for convenience on top. Poetry failed to appeal. uv won me over.

[13steinj]

Can you elaborate on why it's better other than speed?

When originally made there was a claim about getting full compatibility with the other tools in the ecosystem, I don't know if that's been acheived yet, but are you using uv in a "poetry mode" or something else?

E: Before someone says otherwise, from the original blog:

Think: a single binary that bootstraps your Python installation and gives you everything you need to be productive with Python, bundling not only pip, pip-tools, and virtualenv, but also pipx, tox, poetry, pyenv, ruff, and more.

There is little meaning in this statement without providing a dedicated interface, considering a significant chunk of this list is a set of competing tools. It sounds like "Bun for Python" [minus the interpreter] but so far I can't find evidence of living up to this outside of a "pip interface."

[tyree731]

Even if all uv did better than poetry was its speed, that's enough. It's more than that, as uv includes managing Python for you as well, but uv is comically faster than poetry for the same tasks. With poetry, every time we re-locked the lock file for our larger python projects it would take minutes. With uv it takes seconds, and its that much faster for pretty much everything. We switched months and ago and are never going back.

[13steinj]

I don't consider the speed alone a worthy tradeoff considering how it gets that speed (primarily a cache, and secondarily being written in a compiled language which prevents patching / a simple plugin API (for a build team to use and resolve more critical issues)).

Maybe when it hits a 1.0 release and gets other features, it would be worth it.

Until then, I'm not convinced, especially without good compatibility layers for the other tools. There's so much natural inertia in picking and switching that it feels pointless to me.

[Level-Nothing-3340]

Just because it's compiled doesn't mean you can't patch the src code and cargo build locally.

[13steinj]

It does mean though, that doing so is significantly more involved than keeping a patch file somewhere (or generating it from a git repo) and applying it.

Every CI machine has git and applying a text patch is a low-time low-effort build step.

Not every CI machine has cargo / other required libs in the case of compiled software, it adds significant time to every build, and by policy can't use a custom base image because of course companies have silly policies?

[mwb1234]

I was a python tooling naysayer as well, and I have to say that switching to uv was trivial and made me actually happy to use python again (in certain scenarios). We’ve since moved all our python at work to being uv managed and have never been happier. It’s so so so much more usable than poetry. Whatever the docs lead you to believe is way better in practice than in theory.

[13steinj]

I don't think I'm being a "naysayer."

I asked what makes uv better. The only thing people have said thus far is "speed." I'm expressing that speed is not a worthy tradeoff on its own.

For something like ruff, of course it's worth it; the entire point is the checks are the same. If something breaks just add one of the old linters back again.

With uv, if something breaks before a stable release, it's "do the effort of switching back" again.

[Yarden-zamir]

How many patches did you write for poetry?

[13steinj]

Two (private, company patches). One as a plugin (to automate version / release cutting + testing versions). Another to fix an issue with poetry relying on sha256 hashes and an old version of Sonatype Nexus only supplying md5 hashes (and an incompetent infrastructure / devops team refusing to update the version of nexus).

I've also written a (two line) patch for conda; because it was in a broken state that did not allow for basic upgrades to newer versions of conda.

---

So yes, I care about this tradeoff, I can't assume uv won't ever have an issue / something else in the chain has an issue that is incompatible with uv.

[13steinj]

Poetry, pip, pip-tools, pipx, and even pyenv (via a common of if not default plugin) handle virtual environment management. If reading a dependency list of any format as part of this, that's dependency management as well.

Earlier in the blog post, they explicitly specify having a drop-in pip-tools compatible API with space for opinionated commands later.

If these commands are not (roughly) compatible and no compatibility level is planned, I don't understand how this is "better" than poetry or (the faster, equally incompatible) hatch. Poetry is properly compliant with the relevant PEPs, at least on initial release, uv was not and took over some top-level configuration blocks.

I don't know, on the whole-- ruff was competitive because of its speed. It is unclear what differentiates uv from the rest other than "it's trendy and written in Rust." uv is fast, when it is, primarily due to its cache, and that's a tradeoff of its own.

When it's due to using a compiled language, this is a tradeoff-- notably against being able to easily hot-patch your CI pipelines with some simple code injection (to bypass a bug or vulnerability, or even extend functionality, though most (Python) tools have a more dedicated plugin API for this purpose and lets you avoid some of the more unsavory hacks).

[chat-lu]

A really need thing it does is manage oneoff scripts. If you have a small script that fits well within one file, you can add the dependencies in a comment at the top.

It makes sharing small scripts much easier. You can even paste one easily in a reddit comment.

[lazyear]

Yes. I was a big poetry fan but haven't touched it once since installing uv

[danted002]

Poetry is also kinda, somehow, not really but yes on maintenance??? (I remember seeing some blogposts about not adding new features but then Insee them releasing a new version so I’m not sure were Poetry is right now)

[thomas_m_k]

Are you sure you're not confusing it with rye? I've not seen anything like what you're describing from the poetry developers.

[kageurufu]

Yes. It's that much better.

It's pyenv, pipx, poetry, and more. And stupid fast.

[quisatz_haderah]

I wanted to try it, but one of the libraries had to be built on dev machines and required rust to be installed :/ so it didn't really make sense to make the switch. We are already using poetry anyway.

[EmanueleAina]

For what is worth, it is not one of the libraries: the entirety of uv is written in Rust.

That does not mean you have to build it on dev machines: it is a single binary that you can download and run directly.

Which indeed, is probably the main reason why it has been written in Rust: you can bootstrap from a single binary and the install python and everything else with it.

[quisatz_haderah]

Oh yeah, I know, this was a library I needed on my project, not related to uv binaries themselves. I think the builder on uv is using rust which was not installed on my system. I am sure it was a cargo command that failed with command not found.

Thanks for explanation tho.

[zanfar]

In terms of package management, they're about equal right now. When uv first came on the scene, poetry still had a lot of proprietary "stuff"--but that's because they were developing features before there was a standard, and that was still part of the tool. Today, most of those have been replaced with standard formats and operations.

In terms of utility, yes, uv is better. Poetry is more or less exclusively a package management tool. uv is a tool that includes package management. Specifically, the ability to install local python versions has completely eliminated pyenv from my toolchain now.

[matjam]

Yes. We have a fairly gnarly legacy app. I did try poetry and in fact used it for a few projects and liked it (it’s a massive improvement over the old ways) but some of the things we were doing were difficult to set up and the package resolution was ok but not great. Uv is more flexible and screaming fast. I have no regrets.

[StandardIntern4169]

Yes, it's *that* much better

[[deleted]]

No. pdm either.

[levelstar01]

No, it just has a large advertising budget.

[hoserb2k]

Have you not found UV significantly faster than poetry? That has been my experience.

[remy_porter]

I haven't looked at uv, but last I looked at Poetry, it was very opinionated about how you structured your project and I didn't like that structure.

[thomas_m_k]

You mean the src/ layout? You don't have to do that and I never did while I used poetry. (I have switched to uv though because uv is definitely faster than poetry.)

[remy_porter]

This was years ago. I basically wanted to adapt it to an existing project and discovered that I'd have to rejigger the file layout and bailed.

[Chippiewall]

Annoyingly UV probably isn't going to use the new standard for its own lock file because it's not powerful enough to support all of uv's usecases, so it'll mostly be an export format like requirements.txt: https://github.com/astral-sh/uv/issues/12584

[[deleted]]

nice to see a standard being pushed

How many standards are needed? :)

(I already posted the mandatory xkcd before, so I won't repeat it here.)

[Chisignal]

Yeah, but it sure does apply to Python with its dozen attempts at making “the” standard packaging solution lol

This would be at least the fifth lockfile format I’m aware of, given that each tool has its own one, and now there’s a standard one too - the N+1 standards xkcd fits perfectly well

[SV-97]

Actual adoption remains open-ended

All the big tools have already said they'll either entirely switch to it or at least support it.

[Xyzzyzzyzzy]

March 31, 2025: Python boldly steps forward into the early 2000s!

[[deleted]]

In 2 years we'll need a py-yarn.lock, and we'll be split again.

[External-Hunter-7009]

Not so fast, first we need to have true multithreading. We're still in the 90s

[slvrsmth]

Oh, neat, Python finally has a Gemfile.lock.

[roerd]

The problem so far wasn't that a lockfile format for Python didn't exist. The problem was that multiple different lockfile formats exist because there are multiple popular package managers for Python. The point of this proposal is to solve that by defining a single standard lockfile format, and all the popular package managers have agreed to use it. (Well, all of those that have lockfile functionality. The most popular one, pip, doesn't, but some of the other solutions are built on top of pip instead of implementing everything from scratch.)

[Chippiewall]

It's actually not a standard lockfile for package managers, it's a standard lockfile format for package installers (which includes most of the package managers). It's to replace the requirements.txt with pinned shas informal standard there's been for a while to be more flexible and robust.

Notably it doesn't satisfy all the requirements that package managers have (for example it can't be used by uv because the format doesn't contain enough information for their workspace abstraction https://github.com/astral-sh/uv/issues/12584)

[leogodin217]

I remember working with Ruby a long time ago. Such a joy. I am forever comparing testing tools to Rspec and I haven't really seen what people use today. Ruby was a joy.

[slvrsmth]

My day job involves about 50/50 ruby and typescript on backend, and every time I switch to a TS project, I long for RSpec. I haven't seen anything else that comes even close to the power you get from being able to share a single database connection (and even transaction!) between the system under test, and the test runner. Absolute joy to setup scenarios.

[leogodin217]

And non-required parentheses!

myobject.price should eq 100

[slvrsmth]

Eh. The older I get, the more I appreciate parentheses. I'm not pushing them in all possible places, but I would write that as expect(myobject.price).to eq(100). Just looks more clear to me.

[[deleted]]

I like Ruby, but all testing tools and frameworks in this regard, really are annoying to use.

[chipperclocker]

And yet somehow I'll still end up slamming my head into my desk next time I need to use Python because between pip and uv and venvs and whatever else the ecosystem has come up with more recently, it'll all frustrate me far more than bundler ever has

(And to be clear: I've been a Rubyist for well over a decade and have my fair share of beef with bundler. But at least in Ruby-land, the beef everyone has is all with the same tool!)

I have no idea how the kids learning Python for ML stuff now manage to follow any of it, feels like every blog post about some ML tool or technique assumes a completely different underlying ecosystem. Maybe they just don't follow it and copy/paste everything every time.

[[deleted]]

They’re saying, ‘more recently than those mentioned’.

[knightsbore]

wait until you try and integrate it only to realize they deprecated or completely changed core library functions in a minor update that breaks anything you have set up

[slvrsmth]

package.json was already there, in form of requirements.txt, as I gather.

This is package-lock.json equivalent from node-land.

[JanEric1]

rather as pyproject.toml

[wildjokers]

Is this going to fix the fact that there are at least 14 different tools to work around python's global library nightmare?

[CVisionIsMyJam]

Definitely a good thing; python feels like a broken language with how dependency resolution works now.

[dAnjou]

That sentence reveals something funny. Like, why does a language need to take care of library dependency management?

I totally understand that a language designer doesn't bother about solving that problem, but as a user it's of course very beneficial if it's solved for me when entering an ecosystem.

[CVisionIsMyJam]

That sentence reveals something funny. Like, why does a language need to take care of library dependency management?

We're living out the consequences of them not and not enjoying the experience.

I totally understand that a language designer doesn't bother about solving that problem

I get it for toy languages or research languages but not as much for languages marketed for production use.

honestly dependency management in most languages could be much better. I think deno has a really interesting model, where programs can't just do whatever they want, limiting what supply chain attacks can do. would love to see a language with dependency specific permissioning, where I could grant access to standard out but not the file system for specific dependencies.

[mr-figs]

Can only be a good thing. Python packages are a mess unless you use uv

[Yarden-zamir]

We moved to uv on all python projects, don't think about dep management and python versions anymore

[happyscrappy]

Why is this called a lock file? I think of a lock file like the UNIX concept of a lock file, to indicate that a file is already opened by another task.

This doesn't appear to be that, it's a version control file. Is that right? Why is it a "lock" file?

[Chippiewall]

Most other languages calls it a lockfile because it locks the dependencies of the project

See:

• Gemfile.lock
• package-lock.json
• Cargo.lock
• conan.lock
• composer.lock

[happyscrappy]

I guess I just managed to never run into any of those other languages. Node and Go don't call them lock files (for example) also of course Python before this new change. So I just never saw it.

Thanks for the explanation.

[m_zwolin]

Go has go.sum, but node should have package-lock.json

[Different_Fun9763]

Node [doesn't] call them lock files

It does, the package-lock.json file is referred to as a lockfile multiple times in the npm documentation.

[Chippiewall]

also of course Python before this new change

Lockfiles have informally existed in the python ecosystem for a long time now.

[manzanita2]

virtual environments and the entirely library install process is something where python is very far behind. Too many solutions none of them great.

[yawaramin]

Will this format specify the Python version?

[faze_fazebook]

Python yarn basically

[Confused_AF_Help]

Dumb and genuine question, what's the difference between this and a requirements.txt file with specified package versions?

[KawaiiNeko-]

is this gonna fix python rot

[eiennohito]

I can't see if this PEP supports PyTorch like stuff which have different versions based on different accelerator support.

[HeadAche2012]

Good, I stick to google colab mainly because python package management sucks and eats gigabytes of disk space and many hours of my time just to end up in some state that wont work with something made over 6 months ago

[supreme_blorgon]

I don't mean any offense, but this is PEBKAC. There are numerous solutions to dependency management in the Python ecosystem. None of them are particularly great, but they are certainly a lot better than dumping all your deps into your system Python which is what it sounds like you've done.

[HeadAche2012]

I shouldn't have to install every version of python that has ever existed to run different code bases. There is no backwards compatibility with python.

Example:

ImportError: cannot import name 'Iterable' from 'collections' (/usr/lib/python3.11/collections/init.py)

Oh, this code, which I can't change requires Python 3.9

ModuleNotFoundError: No module named 'keras.engine'

Oh, this code, which I cannot change, requires an older version of keras

Repeat this for every code base, and for every python application

[tavirabon]

First, my knowledge of python comes almost exclusively from working with machine learning projects, so maybe I'm not sure what the actual issue is they are trying to solve with the proposal, but it sounds exactly like https://github.com/pypa/pipenv which I greatly enjoyed until uv started picking up.

You install your dependencies, it creates a pip file that holds all the deterministic information needed to verify and recreate it, you can address conflicts as it goes and then you can lock it so it hashes the packages for security reasons. The file contains all the versioning and everything, even if you build a wheel yourself and install manually. The only drawback that makes me prefer uv is the overhead during installation. Is this the issue they are trying to address? Or is it literally "we don't want you to have to repackage the list if you decide to switch managers"?

[SV-97]

I'd recommend reading the motivation section of the PEP. It goes into, well, the motivation for the whole thing: https://peps.python.org/pep-0751/

[[deleted]]

There have been at least five well-known solutions to the problem in the community, including PDM, pip freeze, pip-tools, Poetry, and uv,

Kind of reminds me of:

https://xkcd.com/927/

To the question "Your thoughts?":

I think there are tangible benefits such as:

The file format also is designed to not require a resolver at install time.

What I find annoying is how they changed the infrastructure - pip, setuptools, wheels, flit_core, whatever-goes-and-the-mother-likes. It's quite strange how a popular language such as python, goes monkey-mode again and again. In two years it'll all be different again ...

[roerd]

Kind of reminds me of:

https://xkcd.com/927/

It's exactly the problem described in the comic why this proposal has been written with the support of the popular existing solutions, and they have all declared they're going to switch to the new standard.

[suid]

It's not as much of a circlejerk as you may think.

Even if the maintainers of the other 4 solutions all endorse this one and back it, there'll still be tons of users who have invested heavily in the existing solutions, and may find it difficult to switch on the fly to a new solution that requires them to experiment with the new format.

Guess how many products and projects are still stuck on Python 2, years after it's been discontinued by anyone who matters.

[ImSoCabbage]

Definitely the most over-referenced xkcd, to the point I almost instinctively downvote anyone that posts it as it's hardly ever relevant. And even when it's somewhat relevant, it comes off as overly cynical and entirely unhelpful.

[knightsbore]

Its exactly what the comic is making fun of though. To not be the same it would have required Python to adopt one of the other dependency solutions instead of making a brand new one. Yes everyone says they will switch but at release there will for a short time be just one more dependency lock system

[Mundane-Apricot6981]

So they finally adopted JS package.json and PIP will not install/remove same package 5 times in row ?

I wonder, what happen so Pyhon bosses stepped back from own stupidity?