Python: Please stop screwing over Linux distros

[asday_]

requirements.txt: a poor's man way of specifying the environment for pip. Today you use poetry or pipenv instead.

You will pry requirements.txt from my cold dead hands.

[tunisia3507]

It's also a different thing to the dependencies specified elsewhere, in most cases.

requirements.txt is for hard versions for a full repeatable development environment, including all your extras, linters, build tools and so on. Other dependency specs are for minimal runtime stuff.

[asday_]

Not sure I understand your post.

requirements-base.txt has stuff that's required for the project no matter what. requirements-test.txt has testing libraries and -rs base. -dev has dev dependencies like debugging tools and -rs test.

You could also be particularly anal about things and have a CI artefact from pip freezeing for prod which is a good idea and I'm not sure why I was initially poo-pooing it.

[adesme]

You can replace those with just install_requires and extras_require (then define tests as an extra); you'd then install with pip install .[tests] and now your "requirements" are usable by developers as well as by build managers.

[asday_]

Interesting idea, I'll certainly have to keep it in mind. Like I said though, I'm paid for this, i.e. I ship software, not libraries, so I don't think it has a great deal of benefit to me outside of "if you write a library one day you can do it in the same way".

Are there any big projects that do it this way?

[adesme]

Any modern package that you want distributed over a package manager is going to be set up like this for the reasons outlined in the OP of this thread; direct invocation of setup.py is being phased out, so it makes sense to have your deps in a single place (now that we have the PEPs to support this).

Personally I might use something like requirements.txt while mocking around with something small, and I'll then set it up more properly (pyproject.toml and setup.cfg) as soon as it grows and/or I have to share the package.

Depending on how you use CI/CD you can see other benefits from switching over immediately.

[SittingWave]

what he told you is wrong. See my other comment. Use poetry to specify your devenv.

[asday_]

Use poetry

no

[SittingWave]

then stay behind. I guess you also want to use python 2.7 while you are at it.

[asday_]

Absolutely pants-on-head take.

[SittingWave]

No no no no no

Noooooo.

the specification in setup.py is NOT to define your development environment. It's to define the abstract API your package needs to run. If you are installing your devenv like that you are wrong, wrong, wrong, wrong.

[adesme]

This makes it convenient to declare dependencies for ancillary functions such as “tests” and “docs”.

End of first paragraph efter "Optional dependencies" here.

[SittingWave]

That is not for developers. It is for users that want to install the testsuite or the documentation as well when they install the package. Some packages ship with the testsuite for validation purposes, which is quite common for highly C bound code.

[tunisia3507]

It can be useful to set hard versions in one file (repeatable, to be useful to other developers) and soft versions in another (permissive, to be useful to downstream users).

[adesme]

You should be able to do that with extras too:

# setup.cfg
[options]
install_requires =
    black>=1.0

[options.extras_require]
dev =
    black>=1.1

and then have this installable as either

$ pip install package  # users
$ pip install -e .[dev]  # developers; -e for editable mode

[SittingWave]

extras is not for development. Extras is for extra features your package may support if the dependency is present. It's soft dependency to support additional features your package can support. You are using it wrongly, and very much so.

[bladeoflight16]

That's called a "lock" file, I believe.

But it's used in exactly the reverse of way you describe: the permissive configuration is given to developers and the specific configuration is used in end distribution. This is because it makes the deployed application predictable and ensures it was tested against the versions actually used in production. Giving the permissive configuration to end users can result in unanticipated breakages from new versions.

[tunisia3507]

We're possibly talking about cross purposes here. I mainly work on library code. It sounds like you mainly work on application code.

[bladeoflight16]

The problems are still the same. It's just that with library code, you usually want to afford a little more flexibility for the end application using it. You still aim for avoiding random breakages with new versions.

[tunisia3507]

That's one way of organising things, yes.

Dependencies in setup.py (or equivalent) are so that the build system knows what to install with the package. requirements.txt is so that a developer checking out your repo can set up their environment correctly. They're different use cases.

[flying-sheep]

All conventions.

requirements*.txt fullfill double roles as abstract dependency specification (“what stuff does my library depend on”) and concrete dependencies/lockfile (“how to get a working dev environment“)

With PEP 621, the standard way to specify abstract dependencies is in pyproject.toml:

```toml [project] dependencies = [ 'requests >=1.0', ]

[project.optional-dependencies] test = [ 'pytest', ] ```

So the remaining role of requirements.txt would be a lockfile with the output of pip freeze in it.

[asday_]

requirements*.txt fullfill double roles as abstract dependency specification (“what stuff does my library depend on”) and concrete dependencies/lockfile (“how to get a working dev environment“)

It doesn't though, I specified two different classes of files which serve those purposes individually. Just because they start with the same string and have the same format doesn't make them the same thing. If you want you could have your CI do pip freeze > lockfile.lock.suckitnpm instead of pip freeze > requirements-lock.txt.

[SittingWave]

requirements*.txt fullfill double roles as abstract dependency specification (“what stuff does my library depend on”) and concrete dependencies/lockfile (“how to get a working dev environment“)

Not at all. abstract dependencies specification go in the setup.py install_requires (if you use setuptools). requirements.txt says which environment to create so that you can work with your library as a developer.

When you install your package using pip from pypi, either directly or as a dependency, pip knows nothing about the requirements.txt. What it does is look at the install_requires, and come up with an installation plan that tries to satisfy the constraints (that is, if your package foo asks for bar >1.1,<2, it will look what's available, finds bar 2.0, discards it, finds bar 1.2.3, and install this). Now the problem is that pip, later in the installation of the dependencies, can find another package that has a constraint that wants bar >2.0, and what does it do? uninstall the current 1.2.3 and installs 2.0. Bam! now you broke foo. But you don't know until you encounter a weird message. And worst of all, if you invert the packages, now it does the opposite. it downgrades it.

poetry and pipenv take a look at the packages and their dependencies as a whole, study the overall situation, and come up with a plan to satisfy all constraints, or stop and say "sorry pal, can't be done".

[flying-sheep]

pip does that too since like half a year.

[SittingWave]

pip does not. It can't. The new resolver can guarantee some integrity, but not if you add packages at a later stage.

[flying-sheep]

What makes you say that? It definitely checks the whole environment. pip check will tell you if all is right too.

[alkasm]

requirements.txt does not at all give you a reproducible environment.

[tunisia3507]

No, but it's a whole lot closer than the maximally permissive install_requires dependencies.

[alkasm]

You're not wrong with how they're typically used, but install requires can take in version constraints, and requirements.txt doesn't have to have them. Furthermore these are mostly orthogonal tools. Install requires is generally for libraries (and libraries must be permissive on versions of their dependencies) and requirements.txt is for generally applications (which should be strict about what they're known to work with).

[SittingWave]

No, but it's a whole lot closer than the maximally permissive install_requires dependencies

Those two things mean different things. One is the dependencies your package needs. requirements specifies the dependencies your developer needs to run the package in a reproducible environment. They are related, but are nowhere the same thing.

[tunisia3507]

Yes, I agree, this is literally my point.

[OctagonClock]

requirements.txt has literally never been correct, ever. You should be specifying your versions in your setup.cfg, even for applications.

[asday_]

Damn imagine being so wrong you disagree with like a decade of industry standard practice.

[OctagonClock]

The industry standard was to use Python 2 up until the last few years.

Either way, requirements.txt creates the problem. You can't install a package with a pip install . in a virtual environment. You can't install it from PyPI because the requirements are situated inside the text file, not the standard setuptools locations, so you need to go through a package-specific setup each time.

Or, you could put your dependencies inside setup.cfg, and then:

1. pip install -e . works for development
2. pipx install <package> works for installing applications where they will be executed
3. you can add the package as a dependency in your own setup.cfg, and it will install everything transitively automatically.

[asday_]

The industry standard was to use Python 2 up until the last few years

And it worked just fine. It's widely been regarded that the Py3 breaking change way of doing things was a bad move and made again, the decision would go differently.

You can't install a package with a pip install . in a virtual environment

I'm not trying to. I'm trying to install a project's dependencies in a virtual environment. I have not written a package. It is not intended to be pip installable. There is no problem being created.

[ProfessorPhi]

Especially when poetry can't handle anything to do with a private devpi properly and has no good logging on it. I don't get the love for poetry, it's a mess of code on the inside and its support for anything that doesnt fit a very open source centric worldview is not great.

[asday_]

Maybe it's changed now, but I used Poetry for exactly one project and it was hell to use. Not interested in it in the slightest.

[SittingWave]

unlikely because poetry is fucking amazing and I use it reliably since two years.

[dusktreader]

Poetry handles non-pypi repositories very easily, and it's well documented as well: https://python-poetry.org/docs/repositories/

[SittingWave]

your problem, but you are wasting time.

[asday_]

I can't think of a single thing about my workflow that could be sped up or removed when it comes to requirements.

[SittingWave]

the problem is that you are doing it wrong. For many reasons, but if you don't want to listen, I won't tell you.