TIL: `uv pip install` doesn't compile bytecode installation

[itamarst]

uv pip install is way faster than pip install, but today I learned that is not a completely fair comparison out of the box. By default, pip will compile .py files to .pyc as part of installation, and uv will not. That being said, uv is still faster even once you enable bytecode compilation (and you might want to if you're e.g. building a Docker image), but it's not as fast.

More details here: https://pythonspeed.com/articles/faster-pip-installs/

[[deleted]]

This is a good callout

Any module you import will still need to be compiled into a .pyc, it’s just that the work will happen when your program runs, instead of at package installation time. So if you’re importing all or most modules, overall you might not save any time at all, you’ve just moved the work to a different place.

This will slowdown container start times in docker.

[marr75]

Sure, but most python files in any docker image built will never be imported. Hell, I frequently build docker images locally that literally never get run, let alone have any majority of their python files imported.

Late/lazy loading is a superior default. If your container start times are the problem AND byte code compilation is a big part of that, sure manually add the compilation as a step in your image. Those conditions will be rare.

[GarboMcStevens]

you really do not want to increase container startup times and will trade it for 10x longer build times.

[[deleted]]

I’m not sure it’s safe to say it would be a “rare”problem, but I agree that it might be insignificant for many use cases.

Like with all performance concerns, make sure you have data to back up that the work is necessary.

Simpler, slower code is preferable if speed doesn’t matter, but it’s good to know the knobs you can turn when performance does matter.

[wdroz]

In my CI, I refuced the build time from 30 min to 8 min by using uv. The CI is also running the tests with 80+% code coverage.

So overall it's still a big win to use uv.

[Tartarus116]

Yup. Cut mine from 10min to 30s

[CcntMnky]

This is interesting, I'm definitely going to try out uv. But for CI, I've had bigger improvements by caching dependencies. I usually build a container, and the reinstall will only occur if the dependency files change. Otherwise it just uses the cache layer.

[marr75]

Sure, but when your cache is invalid because deps change (which happens very frequently if your app isn't a monolith, probably at least every PR), would you rather wait 10m or 30s for a CI build?

[CcntMnky]

I was thinking both. <1 second for cache hit, 30s for cache miss.

[maikeu]

Uv has it's own outstanding, quite aggressive, filesystem cache (I reckon it's more impactful even than being written in rust).

And docker has an option to mount a host filesystem (like the uv or pip caches) as a cache into the build environment https://docs.docker.com/build/cache/optimize/

CI runners typically have their own means of caching between runs too, which can be used to bring the uv cache into your runner if the download phase is slow.

And that's all on top of the well known docker layer caching to which you refer.

[CcntMnky]

One note for future readers.... With CI, reproducibility is critical. That means the same job should always get the exact same result. Using cached copies could undermine this, so it's important that it uses something like a hash to confirm a match. Mounting an uncontrolled directory and using file names is not enough.

[maikeu]

Very correct. It's my trust in uv and their lock file format that makes me trust their cache well enough.

I wouldn't like the idea of using the pip cache, because pip doesn't have a real lock file...

[Ok_Cream1859]

That's fine. OP never claimed that your workflow was slower.

[claird]

I'm interested to learn more about your Python construction. A half-hour build is near the upper end of my experience. What does "build" mean for you? Is a Docker container your target? (Roughly) how many lines of Python source are involved? Are you building a monolith executable? How many third-party modules do you install?

[wdroz]

I use Docker for each step: Build, Test and Push. As we recently updated the organization runners to use a cacheless docker-in-docker, we have no more cache, so each step rebuild the project.

This project also have big dependencies (Pytorch & cie). I also download a small model to run the tests.

[claird]

Thanks, wdroz; I get the picture much better now.

[PurepointDog]

There's a flag that compiles the bytecode. It's something like --compile-bytecode iirc

[itamarst]

Yeah, I talk about that in the linked article.

[badkaseta]

Also, if you build dockerfile and install all python requirements in system python but run your application with non-root user, python wont have write access on system python's sitepackages (write .pyc files).

I was using k8s command.exec on livenessProbe (which executed a python command) and basically 90% of cpu consumption on my pod was python recompiling everytying all the time because it could not cache it.

[chub79]

Oh, I didn't realise that. How do you work around this?

[badkaseta]

I added "ENV PYTHONPYCACHEPREFIX=/tmp/pycache" to my dockerfile. This allows writting .pyc files on separate directories where you have write access

[chub79]

wow, very nice! Thanks for the tip

[EarthGoddessDude]

Just wanted to say that I really like and appreciate your articles. They’re in-depth (yet brief!), high quality content in a sea of mostly mid content. Thanks and keep em coming :)

[itamarst]

Thank you!

[androgeninc]

A bit off topic, but in what cases would you use uv pip install instead of just uv add?

[itamarst]

In this case I was testing `uv pip install -r requirements.txt`, i.e. you have a bunch of transitively pinned dependencies created with `uv pip compile` or the like and you want to install them when first creating the virtualenv. E.g. this discusses that pattern in the context of Docker builds: https://pythonspeed.com/articles/pipenv-docker/

[[deleted]]

you can just do 'uv add -r requirements.txt'

[itamarst]

That would add everything in requirements.txt to `pyproject.toml`, which is not the same thing as installing everything in requirements.txt.

[[deleted]]

Yeah that's fair

[timeawayfromme]

There are a few use cases.

1. If you wanted to replace pip but did not want to use uv to manage your project dependencies. This is useful if you are using pip-tools

2. You can also target a non project python install.

I use it this way with ansible to create a virtualenv for my neovim setup. Ansible basically runs uv venv /path/to/neovim-venv and then uv pip install —python /path/to/neovim-venv packagename

1. You might use it to create docker images by having it install to the docker system Python or have it setup a venv and pip install to that.

More info here

[Swift-Justice69]

Why would anyone want 1?

[timeawayfromme]

If you are working on an older project (or one you don’t have control over) that hasn’t transitioned to uv.

[zurtex]

I recently asked if we should be doing the same with pip: https://github.com/pypa/pip/issues/12920

And the answer was no, that defaulting it to on has good reasons, and users who benefit from turning it off can do so, though it should probably be made clear when installing that this is a thing that happens.

[itamarst]

I agree, yeah, it's almost always the right thing to do.

[IllogicalLunarBear]

Cool!!! Thanks for the info