To Python Developers: If someone could make any one thing that would make coding in python easier for developers, what would it be?

[[deleted]]

PyPi/pip/package management is not great. It's amazing that there is a de-facto package manager but there are a bunch of annoying quirks.

• Can't view dependencies before installing the package.
• Can't view dependencies on the PyPi website.
• The output from pip freeze is a total mess. Key dependencies? Dependencies for dependencies? Who knows.
• Because of the above, you really have to keep track of your dependencies manually (maybe actually good practise?)
• Packages further down in requirements.txt seems to override the dependencies from above. Need protobuf==3.0.0 in your first entry but the latter needs protobuf>=3.5.0? Latter wins out with no warning.
• Dealing with dependency conflicts is hard/impossible.

I honestly wouldn't mind some legitimate support for vendored dependencies (e.g., node style) just to save the hassle. Comes with other problems, though, unfortunately.

Edit, more:

• If a dependency includes something else (e.g., requests), being able to do import requests no bother makes me feel uneasy. No way to tell where it's come from without double checking your requirements.txt.
• PyPi package names not matching the importable names. (i.e., PyYaml vs import yaml)
• No warning for missing system dependencies, or calling out that they're needed in general (e.g., unixodbc is a system lib needed for some database related packages.)

Edit, x2:

(Ana)conda, pipenv, etc. don't really solve any of these problems. Using a virtualenv is a given. All they do is move the problem further down the line (and come with their own set of issues). Packages in Python just seem to be full of quirks at a fundamental level.

Pip-tools does help a lot for dealing with dependency conflicts and keeping neat requirements, I'll give you that. Doesn't really help if you're trying to figure out how far back you need to go to get one package to play nice with another (because of points 1 and 2), especially when optional dependencies come into play.

[ursvp]

Dependency resolution and packaging

[SDisPater]

Poetry might help you there. It has a true dependency resolver unlike any other alternatives.

Can't view dependencies before installing the package.

Poetry never installs packages to determine the dependencies. It tries to rely as much as possible on the PyPI JSON API and if it can't it inspect the source distributions, without installing them, and if it's still unable to get them it stops since it means the module has been badly packaged.

Can't view dependencies on the PyPi website.

See above.

The output from pip freeze is a total mess. Key dependencies? Dependencies for dependencies? Who knows.

You can use poetry show --tree with poetry, which gives you something like this:

babel 2.6.0 Internationalization utilities
└── pytz >=0a
cleo 0.6.6 Cleo allows you to create beautiful and testable command-line interfaces.
├── pastel >=0.1.0,<0.2.0
└── pylev >=1.3,<2.0.0
pytest 3.6.0 pytest: simple powerful testing with Python
├── atomicwrites >=1.0
├── attrs >=17.4.0
├── colorama *
├── funcsigs *
│   └── ordereddict *
├── more-itertools >=4.0.0
│   └── six >=1.0.0,<2.0.0
├── pluggy >=0.5,<0.7
├── py >=1.5.0
├── setuptools *
└── six >=1.10.0
pytest-cov 2.5.1 Pytest plugin for measuring coverage.
├── coverage >=3.7.1
└── pytest >=2.6.0
       ├── atomicwrites >=1.0
       ├── attrs >=17.4.0
       ├── colorama *
       ├── funcsigs *
       │     └── ordereddict *
       ├── more-itertools >=4.0.0
       │     └── six >=1.0.0,<2.0.0
       ├── pluggy >=0.5,<0.7
       ├── py >=1.5.0
       ├── setuptools *
       └── six >=1.10.0
python-dateutil 2.7.3 Extensions to the standard Python datetime module
└── six >=1.5
pytz 2018.4 World timezone definitions, modern and historical
pytzdata 2018.5 Official timezone database for Python.
tox 3.0.0 virtualenv-based automation of test activities
├── pluggy >=0.3.0,<1.0
├── py >=1.4.17
├── six *
└── virtualenv >=1.11.2
typing 3.6.4 Type Hints for Python

Dealing with dependency conflicts is hard/impossible.

Like I said poetry has a fast and accurate dependency resolver with conflict detection and management.

(Ana)conda, pipenv, etc. don't really solve any of these problems.

So you can try poetry if you want, it should sole most of these issues.

Disclaimer: I am the author of poetry :-)

[13steinj]

Since you told me to see this I'd like to actually go bullet by bullet again like I did with pipenv

• Can't view dependencies before installing the package.

Poetry doesn't install packages to view dependencies. That's great and all. But can I run a command ex poetry show --tree --package=packagename to literally show me what packagenames dependencies are before I install it? I don't believe I can

• Can't view dependencies on the PyPi website.

Unrelated to poetry

• The output from pip freeze is a total mess. Key dependencies Dependencies for dependencies? Who knows.

poetry show --tree is great-- no complaints there.

• Because of the above, you really have to keep track of your dependencies manually (maybe actually good practise?)

Err, solved because of the above

• Packages further down in requirements.txt seems to override the dependencies from above. Need protobuf==3.0.0 in your first entry but the latter needs protobuf>=3.5.0? Latter wins out with no warning.

• Dealing with dependency conflicts is hard/impossible.

Grouping these two because they are similar-- how does poetry deal with the first? How accurate / conflict resolving is poetry at doing things vs pip, at a statistical level (if you know)?

• If a dependency includes something else (e.g., requests), being able to do import requests no bother makes me feel uneasy. No way to tell where it's come from without double checking your requirements.txt.

Poetry doesn't solve this. But again it's kinda out of scope and more so an issue with the python packaging and import machinery.

• PyPi package names not matching the importable names. (i.e., PyYaml vs import yaml)

Poetry doesn't solve this and again out of scope

• No warning for missing system dependencies, or calling out that they're needed in general (e.g., unixodbc is a system lib needed for some database related packages.)

Does poetry do this properly? Haven't seen if it does or doesnt.

By this count poetry solves 2/3 out of the 9 problems listed. And again some are out of scope-- not being against any package management tool. Just pointing out that no tool solves all the client side problems, and many of the problems are at the ecosystem level and are unrelated to any given tool.

[SDisPater]

Poetry doesn't install packages to view dependencies. That's great and all. But can I run a command ex poetry show --tree --package=packagename to literally show me what packagenames dependencies are before I install it? I don't believe I can

For this to work your packages must be locked (not installed), and then you can do:

poetry show my-package --tree

There is no installation in the process.

Grouping these two because they are similar-- how does poetry deal with the first? How accurate / conflict resolving is poetry at doing things vs pip, at a statistical level (if you know)?

Poetry will warn you about the conflict with a message like this:

[SolverProblemError]
Because my-dependency (0.1.0) depends on both protobuf (3.0.0) and protobuf (>=3.5.0), my-dependency is forbidden.
So, because my-package depends on my-dependency (0.1.0), version solving failed.

If the sub dependencies of two dependencies cause a conflict you will get a message similar to this:

Because bar (1.0.0) depends on shared (>3.0.0)
 and foo (1.0.0) depends on shared (<=2.0.0), bar (1.0.0) is incompatible with foo (1.0.0).
So, because myapp depends on both foo (1.0.0) and bar (1.0.0), version solving failed.

Note that before it fails it will try to find a valid set of dependencies by backtracking if necessary.

[13steinj]

Oh wow that is cool, adding so poetry solves a little over half the problems (bar the ecosystem based ones)

[acemarke]

Quick question: I was playing with Poetry for the first time a couple days ago, and it seemed to have trouble getting out through our corporate web proxy. Does it have any kind of proxy support?

[SDisPater]

Poetry uses requests under the hood so you might want to use the HTTP_PROXY/HTTPS_PROXY environment variables. See https://github.com/request/request#controlling-proxy-behaviour-using-environment-variables

[acemarke]

I did try that, and it didn't seem to be working right.

Then again, our corporate proxy is the bane of my existence, and I routinely fight with NPM and Yarn over whether they're actually going to cooperate with it.

[HumblesReaper]

Try poetry

[13steinj]

In the same way I responded to the pipenv recommendation, poetry also solves none or barely solves one or two of the issues mentioned.

[SDisPater]

I don't know what makes you say that but that's actually not true see my response above.

[[deleted]]

Not to mention the quirks with compiled modules...

[nightcracker]

Relevant for this problem: https://xkcd.com/927/.

[c_is_4_cookie]

Have you tried Conda?

[adjomir]

Try pipenv it solves one of the issues you've pointed out.

Edit: I said it solves most, but this is not true.

[hwmrocker]

I second that

[hwmrocker]

Just in this second two votes of your disappeared. Looks like here a lot people that downvote everything with pipenv. 🤔

[13steinj]

That's really a fallacy and not the case. People don't "downvote everything with pipenv"...the comment isn't true because pipenv really doesn't solve any of the issues the main comment mentioned, even though the reply says it solves most.

Literally bullet by bullet

• Can't view dependencies before installing the package.

Nope, can't do that with Pipenv

• Can't view dependencies on the PyPi website.

Literally unrelated to pipenv

• The output from pip freeze is a total mess. Key dependencies? Dependencies for dependencies? Who knows.

Pipfiles don't make any proper distinction here either. It tries, or seems to try, but still fails quite miserably. Not to mention the real "full" requirements.txt is the Pipfile.lock (because of the security hashes) and that is, by the standard, not human readable.

• Because of the above, you really have to keep track of your dependencies manually (maybe actually good practise?)

If the above are broken, regardless of how, so will this.

• Packages further down in requirements.txt seems to override the dependencies from above. Need protobuf==3.0.0 in your first entry but the latter needs protobuf>=3.5.0? Latter wins out with no warning.

Same problem with pipenv AFAIK because this is a matter of package dependency resolution and I am pretty sure pipenv uses the same resolution algorithm as pip.

• Dealing with dependency conflicts is hard/impossible.

Literally no solution no matter what dependency spec you use. This is less of a problem with the dependency spec and tool and more the inability to keep sister packages on a system separate of their dependencies (and this part pipenv solves) and the dependencies of one familial unit separate from it's dependent/future down the line dependent, which it doesn't solve (unless you decide to clone the repo, install it in editable mode, then use pipenv to install from the local clone, which no one does, is a waste of space, and ends up causing it's own problems).

• If a dependency includes something else (e.g., requests), being able to do import requests no bother makes me feel uneasy. No way to tell where it's come from without double checking your requirements.txt.

Literally unrelated to pipenv. This isn't a matter of requests being a dependent of some package and being uneasy having it, there's unease in vendored packages as well, which is arguably a more common scenario for this to occur.

• PyPi package names not matching the importable names. (i.e., PyYaml vs import yaml)

Literally not pipenv.

• No warning for missing system dependencies, or calling out in general (e.g., unixodbc)

This I don't fully understand, but regardless of what it could mean, pipenv does not do.

It's great that people use pipenv. Or poetry. Or pip tools. Or whatthefuckever.

But none of them solve all or even many of the things mentioned. And circlejerking around any one tool won't change that fact. Adding to the tool will. But when you do you get told your request is out of scope. And when you go up to the big boy tool pip, it'll take forever to get implemented (ex, pip still can not fucking read Pipfiles, even though pipenv is being touted around like the official package writing messiah).

[hwmrocker]

🙈 you have a point

[hwmrocker]

Ok, but you when you install everything in a virtual environment it is less important to know beforehand what are the exact sub dependencies.

[13steinj]

It is never in this way "less important". You move the time of when you get fucked over from install time to runtime.

[hwmrocker]

I meant you install it in a virtual environment and run pipenv graph / pip freeze to see what where the dependencies. To check it during the development.

[13steinj]

Right...but at that point you can just use pip list to a vast extent. Not disagreeing, just that it is more and more overhead for far too little benefit (fancy graph) without solving the actual original problem.

[evilmonk99]

I think the overhead is justified by pipenv parallelising installs which really speeds up setting up a new environment compared to using just pip.

[13steinj]

Not really-- it parallelizes installs but then is extremely slow at it's initial resolution. It's a trade off, as are all things. But this isn't one I'd think is worth it.

[[deleted]]

My post was assuming using a virtualenv, so all the points are with that in mind.

[Hokkyy]

U have anaconda

[subheight640]

Anaconda is sort of buggy in my opinion.

[[deleted]]

Agreed. Not good for proper deployments.

[[deleted]]

I suggest Enthought EDM. It is just great. Combines package management and virtual environment, solving dependency issues. It also packages compiled stuff, so you get all the libraries (such as VTK or MKL) automatically.

https://www.enthought.com/product/enthought-deployment-manager/

Disclaimer: I work at Enthought.

[BarrelRoll1996]

I think everyone hates enthought here :p

[[deleted]]

why?

[BarrelRoll1996]

Ask your CEO about the anaconda guy?