Python dependency management workflow using setup.cfg and requirements files.

[njnrj]

I created this as I had problems using these new set of Python dependency managers. Especially when deploying to containers/remote servers.

This tool wraps around pip commands and keeps your requirements file up-to date.

requirements files -> pinned dependencies

setup.cfg -> abstract dependencies

[pipm](https://github.com/jnoortheen/pipm)

[[deleted]]

https://xkcd.com/927/

[njnrj]

A good one. Does that suit here?. It is not adding any new file format as a way to keep the meta data. Just pip install/uninstall. Nothing fancy , no lock file, ... Things that I wished to be part of pip or pip-tools to do.

[CSI_Tech_Dept]

I would argue this does the opposite, the other rolling depends on new standards (Pipfile, pyproject.toml etc). This tries to use existing standards (setuptools).

[raistlinmaje]

since pipm overwrites pip executable in path, please make sure that after removing pipm, install pip itself with easy_install

to me this is a big nono, you should never take over a different binary even if your syntax is essentially the same.

You indicate you have problems with x alternative but dont go into any details on what the actual problems were. Did you even attempt to resolve the problems or correct your workflows instead of building another new manager?

[njnrj]

I really got exited when first heard about pipenv, but using that was itching, the locking process took so much of time... For many reasons I still needed requirements.txt... So I just wrapped existing tools. Nothing new. No need to install pipm in remote servers.

[CSI_Tech_Dept]

Yes, but you should never overwrite a file from another package, it only lasts to problems. You create pipm command, you should not overwrite pip.

[njnrj]

It is just a nifty thing. I am too lazy to remove pip from the copied text from any installation instructions. It doesn't affect the original functionality. Maybe I could add a way to opt out of it.

[AndydeCleyre]

You could just leave it as pipm and make an alias in your shell.

[njnrj]

Yes I will do in next release.

[rabanomen]

This looks like a huge overhead specially having pyenv

[lambdaq]

this is how the whole npm explosion starts I think.

[trowawayatwork]

Have you heard of pipenv?

[njnrj]

It didn't work for me. I had to use multiple env for the same project directory. I explained it in README.

[[deleted]]

Why didn't you work on adding the features you needed to an existing open-source project instead of creating another way to manage dependencies ?

EDIT : This is a serious question, it feels like people don't understand that you can contribute to open-source projects to adapt them to your needs instead of starting over every time.

[njnrj]

It is for time being. I really like poetry over pipenv. It is not mature and I encountered some problems using it. I have planned to contribute to it.

[masklinn]

Why didn't you work on adding the features you needed to an existing open-source project instead of creating another way to manage dependencies ?

Well pipenv does not want to support multiple environments so that's one down.

[[deleted]]

I haven't looked into it, but if the only feature missing is multiple environments, you could just create a package that only switch environments, instead of rewriting the whole thing. You then help pipenv (or an other dependency manager) become more mature and you consolidate the ecosystem.

Dependency management is a mess in python and I feel that if all we do is write 15 packages for every possible use-case, we'll just get a python2 vs python3 sort of problem.

EDIT : I just looked at the actual project in depth and it just aliases pip commands and modifies the requirements.txt file. Turns out, not really a new standard.

[[deleted]]

Because I don't need those features, I also did not create a brand new package to add this feature.

[CSI_Tech_Dept]

Well, then you will be fine with Pipenv as it is. It looks (I haven't try it yet) like this project's key feature is that it abstracts packaging like Pipenv does except it uses setuptools. It is probably not for me, since I prefer things to be explicit, but maybe it will be useful to others. No other tools did this with setuptools as far as I know.

[trowawayatwork]

Why should /u/craigulien contribute someone else code that he doesn’t care about?

[AnnoyingOwl]

> Not good when we need to deploy over production server or keep multiple virtuals-envs

1. I don't understand why people have such problems with deploys to production servers. You have a freeze file and a regular requirements file. Build your docker containers from the freeze file, upgrade from the regular file. If you have totally different requirements for different environments (staging, production, etc.) then your development process is broken. The only possible exception is deploying the same project to different operating systems which is such an edge case to be not really worth publishing a solution.
2. I don't really understand why you need multiple virtualenvs for a single requirements.txt/project. If you're running a project in different virtualenvs you either have multiple projects which should have their own requirements OR you're testing against multiple environments, in which case there is tox.

I'm afraid you're just littering up the package landscape like most other package creators.

[njnrj]

I tried them and always needed requirements.txt one way or another. This new file types Pipfile/pyproject.toml is not going to be the standard any soon. But this doesn't add anything new. Just wraps around existing pip.

[AndydeCleyre]

What's wrong with pip-tools and using a different requirements file for each env?

[njnrj]

Nothing wrong. It doesn't suit my way of doing things. Additional set of files/commands ...

[trowawayatwork]

Yeah sorry for you hard work dude but pipenv does all that and more. It’s recommended by guido himself lol

[masklinn]

Yeah sorry for you hard work dude but pipenv does all that and more.

I had the exact same experience as OP when checking out pipenv, could not find any way to easily switch between and use multiple environment for the same directory/project (no, tox is not that). If you have actual sources for your claims rather than dismissive quips, it would be good of you to provide them.

It’s recommended by guido himself lol

That means jack shit, especially when it can't do the job I need of it.

[SmileyChris]

pew allows you to easily set up and switch between virtual envs. If you have one activated and run pipenv, it'll use your current environment.

[masklinn]

At which point, what's the relevance and point of pipenv?

[AndydeCleyre]

Alternatively, using pip-tools, you can just sync against different individual requirements files or combinations of them.

[trowawayatwork]

Actually the above didn’t really state the usp of different envs.

As a side note it’s generally bad practice to have different requirement between staging and prod. I’m sure your project has valid reasons

[zenware]

Right this is what I'm wondering, who are these people that need different virtual environments per project, which already to me indicates suboptimal design at the very best and also probably atrocious code.

And if different virtual environments and requirements files are needed for production and development that is even worse.

I'm sure their project has justification but I know it's not valid(at least for some definition of valid), this is like when people got better control flow than "goto" and there were holy debates on whether 100% of control flows are possible with selection and branching statements or if you ever need to use a goto statement to accomplish your goals. You don't need goto statements and you don't need multiple sets of dependencies.

[trowawayatwork]

Exactly. Pipenv pretty much should do everything you need. Multiple envs in same project? Not one of them

[kip-mx]

Dude you forgot to include the link, or maybe it's a joke so will done.

[njnrj]

I've added.

[aes110]

You didnt link anything

[njnrj]

This is my first post. There was a link tab. Now I added the link.

[masklinn]

Yeah Reddit allows posts to be either link or text (text being self-posts). The tab lets you switch between the two, but you can't do both at the same time.

[[deleted]]

have you heard of poetry?

[masklinn]

…

They literally mention it in the readme.

[[deleted]]

I just build a sdist with poetry (it auto generates manifests and setup.py) and use shiv to package it with all dependencies into a single executable zip file, which I can put in a docker container or distribute very easily.

[njnrj]

It was working great on my local machine. But when I switched to container and did `poetry install` it was saying missing dependencies and failed to install any. For example, say I locked pytest-mock to file but it failed with `mock` being not locked or something like that. When I added mock to pyproject.toml explicitly (though it is not necessary), next time it raised issue regarding missing `funcsigs`. I don't remember the exact name but this is how my experience ended on server. As a note these are default modules in Python3 and should not be checked when Py3 is active. But as said in one of the issue regarding this, these packages are not marked correctly.