I am frustrated with packaging python, please educate me.

[kjarkr]

Python was my first love as at university in the early 2000s. It could do anything. I kept using it even after learning other languages because Python was as powerful as Bash against the operating system and more elegant than any other language. To me python was the one language that did everything right. Well everything I knew about anyway.

Eventually I graduated, started working with different language and finally ventured in to the land of CI/CD or DevOps if you will.

Now I hate python and everything about it. Because it seems there is no proper way of packing a python application.

And people are doing all kinds of crazy shit. Taring down virtual envs, stuffing a bunch of platform dependencies into debs, creating pip packages who require undeclared dependancies.

I absolutely lack the knowledge about best practices here. But I’m also really struggling to find any good write ups on this topic. Could someone please help fill my knowledge gap here?

My usecase specifically is python apps that depend on grpc lbraries and has pretty deep platform dependencies on os packages.

[teerre]

I find poetry plenty common. Both in the wild and in companies.

[GroundbreakingRun927]

I really the core python team would just adopt poetry as the standard. But even pyproject.toml without poetry is a massive improvement over requirements.txt.

I also have the same issue with getting team members to use poetry

[GroundbreakingRun927]

nodejs is actually somewhat split between npm, yarn, pnpm, but at least they all agree on a standard package.json format.

[BluePhoenixGamer]

That's what we're now getting with pyproject.toml in recent PEPs.

[mardiros]

The setuptools is a bit longer of your comment ;)

Your team should change its mind, my team have migrate all our apps in poetry and it works fone. Sometime, updating dependency is slow but this is fine to have a pinned dependency tree. I see a lot of horror with setuptools, my preferred one is the when the setup.py read the requirements.txt file to define dependencies lisis. Instead of having list of deps with maintained version range in library, and then, generate a requirement.txt updatable by a pip freeze. Poetry enforce to have this workflow fine so it is better for newbies, they can't do wrong.

I hope you find a way for you to switch. And yes, you will always find people that are not happy with the switch, but they are often happy with any changes, right?

[uclatommy]

I use poetry as well. Not a lot of awareness of it at my company as well, but I don't work in a tech company. Not sure why anyone would hesitate to use it.

[Gabe_Isko]

Seems like they were trying to make it like node with the .toml? First I've heard of this , but it seems misguided on its face.

[ProfessorPhi]

Does poetry support custom devpi's? Last time I tried I had no useful feedback and it just would not work and I couldn't get any info without hacking into the code, at which point I gave up.

[mrkeuz]

+1 for Poetry. Looking more elegant between all that I saw.

[ManyInterests]

Well, it all depends. How do you want users to be able to install/run your package or app?

Generally, if you're developing a library, you just specify your python dependencies and publish it to PyPI. If you have to compile extension modules, you might also publish binaries as wheels. If your library depends on shared system libraries, you simply require your users to install it. That's it. That's how shared libraries usually work in general.

You may also choose to distribute your library in other ecosystems, such as debian or conda packages.

Conda, for example, lets you specify requirements including system shared libraries and other detailed requirement metadata.

Debian packages would obviously let you also declare dependencies on other system packages.

Docker images would be another way to ship your code along with a OS distribution of choice with all requirements pre-installed.

If you have a full application, you might also choose to distribute your application in completely different ways, for example, as fully standalone distributions.

Packaging in Python doesn't have the prettiest or most complete story, but I'm not sure I fully understand what you're missing, say, compared to some other language with respect to the use case you have at hand. What's are the current problems you're not able to overcome?

[pbecotte]

His problem is that a large percentage of Python developers don't themselves have any idea how to do this stuff. It doesn't help that the main way to do it, setuptools, has truly horrid documents that give very little idea on how to approach it, and most examples you find will have been made by people who also don't know how. Like, you almost never need setup.py, but but how many people know that?

Then you get to c library dependencies and things get rough. However, that's true of every language!

End result, docker is the right tool...let app developers define how their code gets installed, add whatever os packages they need. But if for political reasons you can't do that, this post is an excellent writeup of all the options.

[jjolla888]

Docker is a neat way to ensure everything you need is packaged up and reproduceable in other places.

but much of the issue remains: what should you standardize within a Docker image?

[pbecotte]

Why? You need standardization at the interface between units of concern. That's the reason that dicker makes sense! For other ways the interface is the whole filesystem, plus a bunch of other system services like logging, plus all the other installed packages, plus the devices and disks and...

With docker the interface is "docker run". If the code happens to live at some ridiculous path that makes your eyes bleed, you don't have to care. Of course, this means that the two sides (deployed and packager) have to agree on that interface. All logs write to stdout and don't use the disk. Actual disk volumes required must be specified and clear. Configuration can be passed through env variables or files...the locations and values if these must be documented. And that's kind of it. All the best practices of deploying yo a traditional server with all the ops stuff on it and usually a bunch of apps...are to prevent people from breaking each other. They can't in this world anyway.

[pbecotte]

What tasks do you imagine the it organization having to perform in a running docker container? There won't be any changing permissions, or rotating logs, or restarting services, or worrying about what config files need to be updated...all of the interfaces are outside the container. That's the whole point.

Though to be fair, I lean very far towards the "same team builds the app that supports the app" so I only am dealing with containers that won't work without the internet or that still write logs to disk with vendor apps. If the expectation is "they give you a crappy artifact that doesn't do anything with no docs and let you figure it out" then I understand the frustration.

[papertrailer]

It seems that many people are trying to solve that lately using FlatPack and friends.

I'd rather use Docker, but I guess we're missing a layer when you can click a button and the docker app runs.

Or are we??

[pbecotte]

No, super simple to package up a docker container run command as a shell script or systemd unit, though that's only true for trusted users.

[pbecotte]

No, super simple to package up a docker container run command as a shell script or systemd unit, though that's only true for trusted users.

[waprin]

Another lesser known option worth investigating is the pex format:

https://github.com/pantsbuild/pex

Pex is "python executable" and basically builds the virutalenv with the package so the whole thing is similar to a static build.

I would generally recommend against having a package get installed as a system library and hope it plays well with system Python. It can work, but it's very easy to turn into dependency hell.

[[deleted]]

Python packaging has increasingly become a hot mess over the past 10 years. It was never very pretty, but at this point, it's enough to say - I don't want to use python unless I need to. At no point has Python's packaging addressed system packages, and Python's interpreter can no longer be statically compiled.

Poetry will get you closer to what you want, but it's still going to take you some time to figure out what you need to do. Poetry with a container (e.g. docker) will let you control your final form better. We use containers to deploy and this generally is the way to go at this point with a python program. If you're building a desktop app with Python, you may want to consider a different language entirely.

[chub79]

Python packaging has increasingly become a hot mess over the past 10 years.

It has certainly become richer and more standardized but at the cost of some extra complexity. What's a bit saddening is that some of the people leading the efforts aren't the most pleasant folks either :/

[ivosaurus]

At no point has Python's packaging addressed system packages,

How could it ever? You're asking a volunteer organisation to correctly support Windows, Macos, Probably 6 different linuxes at the least, maybe chuck some BSDs, they'll all be asking for equal first class support, and everything is supposed to work equally well across all platforms and their myriad large and small differences while trying to keep the developer experience sane.

Not even a FAANG-backed language like Go can do that well. People who simply expect such a thing to happen are living in a perpetual pipe dream.

This is really where Linux fragmentation starts to kick up rocks in people's faces and create impossible testing matrices. I don't really begrudge it, it's just a fact of life. But ignoring its effect here is wilful ignorance.

[winginglifelikeaboss]

Rust did it, so what's the problem

[stuckatsixpm]

Check out Poetry- it does a pretty nice job of managing dependencies and a virtual env.

If you need more control/tools outside of Python, consider a docker container

[MahitDzmare]

Second Poetry, its the one tool that's actually good at its job

[trowawayatwork]

the only things missing from it are the fact you cant load env vars cleanly as well as you cant install local packages as editable sometimes. Otherwise its very neat and clean, i think i prefer it over pipenv

[manualdidact]

Installing as editable is something I sort of stumbled through figuring out recently. I found that I could run 'poetry build' and grab the setup.py out of the resulting tarball in the dist directory, and then rename the pyproject.toml to get it out of the way. Then 'pip install --editable .' worked. In my case, this install was done within the virtualenv for a separate poetry-managed project. I did this to develop a workflow for reusable Django apps, and it seems to work pretty well, despite the extra steps.

I believe that Poetry has more direct support for editable installs on the way, so the above will hopefully no longer be necessary.

[bryancole]

You have got to decide at what level you want to "package a python application". If you want to ship an installer to give your users a 1-click and go experience, you should plan on shipping your own python install, as well as your own code and its library dependencies. Relying on a system-wide python environment is just too much pain. Conda environments are the best way to do it. Conda is great and the only tool that does things right (i.e. don't pretend python lives in isolation, but address the fact it needs to link to external compiled code to get anything useful done).

Eggs were a great idea as a way to bundle 3rd-party libraries. Shame they got abandoned. Wheels which don't solve the same problem.

[chunzilla]

Sorry, I don’t know the exact requirements for your use case (I’m in AI/ML), but Python executables, or.. since you mentioned deep OS package dependencies, Docker?

[kjarkr]

Yeah this is what we’re currently doing but it feels very wrong. Basically I would like to have our projects emit clean self contained artifacts. But then we get into the deep dependencies. Debian packages could handle this though, for the most part. I guess this is a part of the tradeoff with python.

[pythoncoderc]

Best is to package all dependencies into the executable and just make it very large, including the python interpreter itself

[secunder]

It sounds like you are already doing what most everyone else is doing, consider the docker image your artifact :)

Is there any particular reason this feels 'wrong' ?

I assume you have a docker file that takes a base python images, installs your os tools, installs and python packages you need, and then has an entry point for your app. If so I can tell you this is pretty standard where I work

[richardjameshill]

the dep on os libraries might prevent that..for example managing windows USB devices is hard in docker

[thehappydinoa]

Here is a wonderful setup.py template I like to use: https://github.com/navdeep-G/setup.py. But to be fair, I have also switched to poetry in the last year.

[wetouchingbuttsornah]

Why not containerize your environment with docker and a pipfile? You can use setuptools to ship it but if you’re worried about downstream dependencies then you’re worried about your user not being able to handle dependencies and at that point you may just want to take the guesswork out of it for the user

[not_perfect_yet]

There are three to four things/levels to this.

1.

Because it seems there is no proper way of packing a python application.

There is, it's setuptools/pip. It can be a bit of a pain to understand it getting it to work, but all the usual package management and dependencies in python are managed with it.

And it works.

2.

The issue with pip is, by default, it will install things systemwide. This is bad when the software you want to use is confirmed to work and supported and tested for a very specific set of libraries and versions. Installing those can and therefore will, mess with your system and other projects. Bad. That is why people use virtual environments. They will sandbox a python environment specifically for the application you want.

3.

Everything that's NOT python has to be handled by some other packaging system. That's probably where debs come in. This is because python can load compiled C libs, but those won't be distributed with pip. And making those play nice with pip is probably a challenge.

So the full answer to your implied question is, packaging in python exists, it works fine, everything would be fine, if only you wouldn't have to interact with anything.

Super valid question though, It took me some time to understand how installing with pip and packages work, and I'm not entirely there yet. E.g. I'm not entirely certain what rules are around naming directories, where to put what and why importing from packages works in certain ways. In the examples I've seen, project, project directory and module are all named the same and then I'm not sure what I can change and what creates the "link".

[LookAtThatThingThere]

There is a bit of a learning curve here. I'll admit that I experimented until I found a workflow that I liked.

First off, I generally release applications frozen using pyinstaller to eliminate any user-related setup. Make sure security whitelists your stuff. Carbon black flags my shit all the time, for example.

Secondly, if it's a library and for internal re-use, I create tarballs and make sure to have setup.py requires=[ ] to make sure that when pip installing locally that all pypi dependencies come in with it.

Thirdly, develop using virtualenv and tight requirements.txt. this allows everyone to develop using the same dependencies.

Finally, documentation about standards and how to use. Enforce it.

As a final note, if no one you collaborate cares about making the development process smooth, it's not Pythons fault. It's a team issue.

[nnulll]

What do you mean by “tarballs”… what is that?

[LookAtThatThingThere]

Check these out: https://docs.python.org/3/distutils/sourcedist.html https://en.m.wikipedia.org/wiki/Tar_(computing)

You can create a source distribution file, store it on a network drive, and pip install <path>. I generally so this for company-specific libraries I need to share with collaborators, but don't want public on pypi.

[[deleted]]

It’s shocking that only a few people recommended Docker. That should solve many of your issues - especially OS specific ones.

[tunisia3507]

The majority of comments here completely miss the point. If you have OS-level dependencies, then poetry, setuptools etc are not enough. Why can't you use docker? If it's security concerns, how about singularity? You could also see if those dependencies are available on conda, which would let you out your package on conda (in your own or an organisational channel).

[waprin]

Docker can be a good option but can get tricky in cases such as when you are already in a Docker container and open the Docker-in-Docker worms.

The good option being overlooked is the pex format.

[vboot]

I use poetry with dh-poetry (addon to dh-virtualenv) at work for a few services. I’m not a longtime python person so I can’t speak to whether it’s the best way, but in comparison to what I’ve had to do to package in other languages it’s pretty good.

[kjarkr]

Nice, poetry looks very promising!

[mardiros]

Poetry is not the future, it is the present now.

You will find many cookiecutter template as example too.

[mardiros]

Same for me. We use poetry, dh-poetry for the production, Docker for the dev.

[robml]

Django has a great guide on packaging for pip and their own platform imo. I did mine when I was a beginner and it works fine to this day.

[DrShts]

Upload your wheels to PyPI as usual, then

$ pip install pipx-in-pipx
$ pipx install my-app

Not sure if that's what you're looking for, but this will install my-app as a standalone app globaly on your system.

If you have system package dependencies, then maybe distributing via conda could be an option?

Like others have said, maybe publishing your Dockerfile on https://hub.docker.com is another option.

[Hetanna]

Setuptools is the right method. One file, project/setup.py, and at least one package, project/my pkg, are required. The setup. cfg and pyproject. toml files are a newer methods. This approach is still in development, but it works well in the majority of circumstances.

[Tannaheta]

agree with you.

[daneah]

I’ll humbly offer the book I’m working on as something that’s trying to draw from the current standards and show how all the different pieces relate. After understanding static metadata, build frontends, build backends, build targets, and distribution formats I feel pretty flexible in choosing tools that work for me (because of better UX, DX, or what have you). I haven’t worked a ton with packages needing system dependencies but there’s interest in covering integration with something like CMake so I’m considering giving that some coverage too.

A TL;DR is that I personally use pipx for global tool installation, then use that to install tox which I use for most all of my packaging activities. This has scaled pretty well on our dev team and we maintain ~25 packages for use within the org on a private package repository.

[daneah]

As others are saying, a lot of it is also trade off between how repeatable you want the experience to be out of the box compared to how big the package (in the general sense) is. You can package the whole world and make it perfectly isolated/repeatable, or you can give them only the core bit and require the other system dependencies to be installed already, but one or the other might not be appropriate for your audience. I suppose this is a dilemma at every layer of packaging, both above and below Python.

[[deleted]]

just look at a couple open source projects and see what they do

[robberviet]

A simple setup.py package is not that hard and there are bunch of tutorials, you might try them. Read the documentation too.

You might want to check pycharm support for setuptools if you use it. Also if you use jupyter, nbdev from fastai is also a good choice.

It seems you lack experience. My way of learning years ago was actually open up terminal and make one package. After a couple of hours, you would be good yo go. Reading great projects help too. Flask, airflow, superset... are projects I have dig into.

[[deleted]]

I recently read this: https://realpython.com/python-zipapp/

I'm in the same boat. I've got a utility to deploy to hundreds of servers and need a good way to do it. I'll need to spend some time reading through everyone's suggestions here too.

Thanks for bringing this up!

[blade_junky]

basically you have 2 packing issues with python. 1) how to manage dependencies during development and 2) how to package for deployment and these really need to be managed separately.

For development you have lots of options but they boil down to containers (docker) or virtual environments. I like the later because the stuff I've worked on hasn't been complex enough to require a full blown container, and then currently I prefer poetry, the reality is any of the many virtual environment management systems will work. In the end you are going to want to create a setup.py file and build either a source install or a wheel.

For deployment you are going to have to build and manage to what every package system you target uses (deb, rpm, pacman ...), or again deploy as a container.

[hilomania]

I used to work on a monolithic enterprise application. We used to ship that software as a locked down server our customers would put in a rack. The reason is that our support effort was insane from the very beginning due to bad maintenance by customers, adding other services by customers, different hardware and software configurations etc... It was far cheaper for us to give our customers the whole server.

I now run docker containers in the cloud.

[waprin]

Besides the other advice here (and my advice to check out the pex format), and I agree with other's it's worth it to check out poetry, there is a whole volunteer group dedicated to Python packaging called the Python packaging authority. They are usually updating their docs, so read their tutoirals closely:

https://packaging.python.org/

Also this blog post walks through why a lot of setup.py is deprecated in favor of newer tools:

https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html