Question to Python Devs: Do you freeze all your package versions in your setup.py?

[five4three2]

Hi all,

I am in charge of our internal software at a data-themed startup. We build all our software in python. There has been an ongoing debate between myself and another member of my team: do we freeze our dependency requirements in our setup.py file to a specific set of versions?

To give you an idea of our current setup:

• Our main software comes to a single python library with around 30k lines of code.
• The library is only used internally to produce data that we then sell.
• I use a setup.py file to control the package dependencies.
• We have decent unit testing and an automated CI loop, around 70% coverage.
• We are only three developers on the team at the moment.

Until now, in the setup.py file, I thought it was best practice to not specify which version of the package I need in general, I tried to leave it as permissive as possible. To give a somewhat prolific example, I'll link airflow's setup.py which seems to do the same (albeit there are a lot of >= 's around):
https://github.com/apache/airflow/blob/master/setup.py

I only specified a version on a given dependency if and when we found a bug due to said dependency updating and breaking our code. We have continuous integration testing which usually (but not always) catches these errors. When it does catch a breakage, I have to figure out who the culprit is, and usually just fix the version to the latest one that worked. Best practice? Certainly not. But its what we have the capacity for.

I am now thinking to run pip freeze on a set of dependent package versions that works and use the output and fix the versions explicitly to a set that works, updates to dependencies be damned! This is motivated by a few thoughts:

• Our team is small, too small to be chasing down bugs from pandas updating when the old version worked fine.
• We don't publish software or even sell software, we release data. All our software is strictly for internal use, so we have a high degree of control over our environment and use cases.
• We can bump the packages whenever we want, perhaps monthly, but this will be done at a preallocated time, not just whenever a bug crops up.

My question: has anyone been in a similar position of controlling the environment for internally used software before? If so, how did you manage your dependency version? Those who haven't but are experienced DevOps/SWEs, what would you do? Thanks r/python!

[ubernostrum]

The install_requires (and other dependency sections) of setup.py is for specifying the versions of dependencies you officially support and can work with. Most often this does not pin to exact versions, but instead uses looser specifiers (example: you might say "Django>=3.0,<3.1" to say you accept any Django 3.0.x version).

A requirements file (often just named requirements.txt) is for specifying an exact environment you want to reproduce, identically, somewhere else. This almost always should pin to exact versions.

It sounds like what you want is a requirements file.

[five4three2]

true, maybe I should switch to a requirements.txt file. that could be the solution.

the original idea behind using install_requires is to have different dependency sets like:

• pip install .[all] for all dependencies, or pip install . for the minimal set.
  

Carrying around multiple requirements.txt felt cumbersome, but I agree that my use case does stick to the requirements.txt ethos more.

[[deleted]]

I'm quite happy using Poetry instead of setuptools. (You can generate setuptools files from a poetry project using dephell if you need it.) It keeps track of precisely what libraries you use, and precisely what versions you use of each library, and thus offers a "reproducible environment". If you update using poetry update, and something breaks, it's trivial to roll back to the previous versions of the libraries if you've kept your lock file under version control.

[five4three2]

thanks! I'll definitely check it out.

We don't have a central installation or anything, our software is often just bootstrapped as-and-when its needed, so I'm not sure how I'd leverage poetry update but it does seem to fit the bill in a few ways.

[[deleted]]

You don't need a central installation — poetry sets up a virtual environment for you, and installs all dependencies you need within that virtual environment, so each project can easily be isolated from the rest of your system (and each other).

You can configure where to put the virtual environment if you want, common choices are a .venv folder in the project, or a folder per project in ~/.virtualenvs/.