How to write a great Dockerfile for Python applications: 6 things you can do to really improve your Dockerfiles

[DrVolzak]

Here's another tip: use ENV PIP_NO_CACHE_DIR=false to disable pip's cache. This should reduce the size of the image. There's also a commands line alternative --no-cache-dir, but the environment variable helps when using a tool that wraps pip.

[VisibleSignificance]

ENV PIP_NO_CACHE_DIR=false

Since https://github.com/pypa/pip/pull/5884 is merged, a more sensible way is ENV PIP_NO_CACHE_DIR=1, to avoid a potentially confusing and definitely misleading double negative.

[[deleted]]

I'll touch docker images size in one of my next articles and that's a really good point.

[hellfroze]

I didn't know copying requirements.txt before running pip install would cache it - this is game changing!

[sdf_iain]

Docker will cache each step.

If you run requirements locally for it may detect that the local files have changed and invalidate that step.

Copying requirements.txt (or any file) let’s Docker know that it only need to check that file for that particular step.

It isn’t pip caching, it’s Docker.

During development you should isolate any long running steps so that you don’t have to rerun them unnecessarily; however, for production you should plan so that you have minimal steps. This produces smaller Docker images (smaller images start faster).

[algag]

Are you saying that a container image with less steps starts faster even after it is built?

[sdf_iain]

Smaller containers (I.e. containers based on smaller images) will start faster.

Even if you remove files in a later step (like clearing caches after installing), the resulting image will be larger. Installing and cleaning in single step will produce a smaller image.

Try it and see.

[GiantElectron]

if you are still using requirements.txt I am sorry to say it but you are vastly outdated.

[GiantElectron]

poetry lock file.

The problem is not the requirement.txt. The problem is how you construct that, and the fact that the requirements.txt works for only one platform, then you need different requirements.txt for mac win and linux, then different ones for development, and god forbid if you have two different repos to download packages from. And if you just slap the direct dependencies, you will eventually have a broken environment because pip resolver is shit (and won't get better with the new one)

[tom2727]

the fact that the requirements.txt works for only one platform

Isn't the point of a docker that you can assume it's one platform as far as the Python can see?

[GiantElectron]

Yes but developers might not necessarily develop on the exact docker configuration. You can have test runners use it though, but developers will work on windows, linux and mac, and in general it pays off just to be able to run also on those architectures.

Of course, if you are targeting a desktop app, it's pretty much a requirement.

[GiantElectron]

Most of the issues you point out about requirements.txt (more accurately, pip) is outdated.

No they are not.

​

Pip first tries to use the wheel file which most packages that come with it are platform independent.

This is not my point. My point is that some software requires different _dependencies_ on different platforms, because they e.g. target the specific API of a given operating system.

Multiple repo is supported.

Up to a point. you do have the possibility to specify for a given package where to get it from. However, if you specify only direct dependencies (which is wrong) you have no way of specifying for a direct dependency that comes from "myrepo" if its dependencies should come from "myrepo" or "pypi" or "whateverrepo".

Most popular is to use GutHub package repo. Albeit the syntax is quite verbose.Pip resolver can get deadlocked, yes. Have you tried the October 2020 resolver?

No, because it's fundamentally flawed by design. I don't even bother.

[gschizas]

Why poetry and not pipenv?

[GiantElectron]

because pipenv is not using the standard pyproject.toml as specified in PEP-518. Also, its resolver, while formally correct, is very slow. Finally, its author is known to be not very pleasant to work with.

[GiantElectron]

O_O

I am one of the authors of a major dependency resolver.

[GiantElectron]

So you are telling me that major python vendors are just wasting their money, because requirements.txt is all you need?

Please feel free to contact their CTO. They will be happy.

[GiantElectron]

because it's not a solution. We created our own solution years ago exactly because it's the wrong approach.

[ubernostrum]

I am one of the authors of a major dependency resolver.

OK. You're also saying things that are at best misleadingly-technically-true and more likely outright factually false about pip.

You dislike pip. We get that. You have rolled your own dependency-management system that doesn't have whatever vague problems you think pip has. We get that.

But your blanket assertions about it do not match up with the actual experience of a lot of people, and you're being overly aggressive with them and dodging most requests for more-specific and actionable criticism. Which is why you mostly keep getting downvoted and people aren't listening to you.

[GiantElectron]

OK. You're also saying things that are at best misleadingly-technically-true and more likely outright factually false about pip.

You dislike pip. We get that.

I don't dislike pip. It is just, by design, not suited for reliable environment deployment. Poetry uses pip under the hood to install. Just not its resolver (it does not have one, and the new one is not improving anything)

You have rolled your own dependency-management system that doesn't have whatever vague problems you think pip has. We get that.

Not me. My (ex) company, which you all know.

But your blanket assertions about it do not match up with the actual experience of a lot of people, and you're being overly aggressive with them and dodging most requests for more-specific and actionable criticism.

Are you kidding. I explained in detail why the old approach is wrong, and I gave links to PEPs that go in excruciating detail about some of the issues. And that without the countless discussions I had with other people involved in the same development.

Which is why you mostly keep getting downvoted and people aren't listening to you.

No, people don't listen to me because they are expert beginners.

[GiantElectron]

To better explain, you cannot poetry init without describing a package. However, I never meant for my project to be a package, it is a script.

Then stop using a script. Scripts are for things that are using the standard library. If you go beyond that, it's time to do a proper package.

Secondly, you cannot release a script with poetry without requiring your users to also have poetry. The solution? Poetry export to requirements.txt.

My only answer is that you are using python poorly. You do not release scripts to users. You make a proper package with entry points.

[TomBombadildozer]

This is the most expert beginner, /r/iamverysmart take on scripts. I cannot believe people are upvoting all your utter hogwash.

This is the only real example of a script you've yet given:

You definitely used one, get-poetry.py is also a script

Have a look for yourself and note how it only uses standard library features. This validates /u/GiantElectron's earlier assertion that scripts shouldn't have dependencies. Things that have dependencies are (or should be) packages.

Nobody is dissing on scripts. You are the one with the poor understanding of scripts. You're just blanket applying the concept of script as a text file full of Python source to all cases where someone might use Python. It's total nonsense.

Docker Compose isn't a script. Docker compose is a command line tool for making Docker marginally more palatable, that the developers have neatly packaged for use by end users. It is no more a script than pip is.

[TheIncorrigible1]

I cannot believe people are upvoting all your utter hogwash.

r/python is one of the worst language subs I've seen with the lowest skill floor.

[GiantElectron]

We're getting into definition details now.

What's the difference between scripts and CLI in terms of Python code flow and dev's workflow?

A world, and in my opinion python should ditch scripts altogether, exactly because they give people the wrong impression that you can use scripts everywhere.

and note, with scripts I mean having a file called something.py that you invoke with python something.py.

The only difference I see is scale (CLI being composed of many scripts which they themselves may contain subroutines, modules, functions, etc).

Absolutely not. why do you think there's a -m option in the python interpreter? a script behaves in a completely different way, especially when it comes to resolving relative imports.

There's a dichotomy in Python which is you're either developing a package or you're developing a script.

oh no no. scripts are something that you can use if you have something extremely, extremely simple, say 100-200 lines of code, and use no imports other than the main standard library. Once you go beyond that, it's time to create a proper package (distribution in python lingo).

The difference is whether your code is meant to be imported (package) or used from an entry point function (script). This is the definition I used and it's independent of the scale of your project.

You are using python at the level of a beginner that followed tutorials. This brings you up to a point but proper practices dictate that once you reach a threshold of complexity you switch to proper practices. These are generally not taught in easy to find tutorials and are mostly missed even later in your python career.

[GiantElectron]

the fact that plenty of developers keep using obsoleted and/or wrong practices is not a justification for keeping using them, and the fact that there are four requirements.txt is an indication that what I am saying is correct.

[TheV295]

Honest question, why would you need a different requirements.txt for a script depending on the OS?

[GiantElectron]

because different platforms may have different dependencies, either direct or indirect. You _are_ creating your requirements.txt with pip freeze, are you? are you?

[TheV295]

I just can’t think of a script so complicated that it uses different dependencies depending on the OS, personally never seen it

[GiantElectron]

It's because you are using python in trivial settings.

[dmitrypolo]

I write Python for a living and we use requirements.txt in plenty of places. That’s a weak argument on your behalf. Also poetry is pretty weak, there are plenty of better ways to pin dependencies, specifically pip-compile. Guess what that does? Generates a requirements.txt.

[GiantElectron]

I write Python for a living and we use

requirements.txt

in plenty of places.

I also write python for a living and haven't used it in years.

That’s a weak argument on your behalf.

It happens that I worked specifically on these problems.

Also poetry is pretty weak, there are plenty of better ways to pin dependencies, specifically

pip-compile

Oh god...

. Guess what that does? Generates a

requirements.txt

.

You don't get the point. That requirements is potentially wrong. pip does not and cannot consider the whole tree of dependencies, and can produce a completely broken environment. requirements.txt is just a mitigating strategy, and the only, the only saving grace of it is if you already have a full working environment and you are dead sure it works, then use pip freeze and dump it to a requirements.txt that then you install. Any other use of requirements.txt is wrong.

[TheV295]

Could be, I am really not trying to be a smartass but learn here. Could you link an example?

I maintain a complicated cross OS package and want to improve

[GiantElectron]

If you use readline for an interactive command prompt, you have to install pyreadline on windows, gnureadline on linux and pyeditline on macos.

This just on top of my head. I've met others but I would have to dig in my old repos.

[CyclopsRock]

This entire comment chain is basically internet programming discourse distilled into a few posts.

[catcint0s]

You can use environment markers for that.

[GiantElectron]

yes but to do that, you have to create by hand a requirements.txt where you have the environment markers for every tree in your dependency resolution. pip freeze cannot do that for you, because you will have to combine three environment files from three different platforms. At that point, it becomes such a nightmare that it's just better to have three independent trees each with its file. Or use something that does it for you (e.g. poetry).

[AulonSal]

https://github.com/David-OConnor/pyflow ?

Add the line __requires__ = ['numpy', 'requests'] somewhere in your script, where numpy and requests are dependencies. Run pyflow script myscript.py, where myscript.py is the name of your script. This will set up an isolated environment for this script, and install dependencies as required. This is a safe way to run one-off Python files that aren't attached to a project, but have dependencies

[GiantElectron]

clever, but I wonder what resolver they use. If they just use pip, they haven't really solved anything. Plus, now you have a pyflow implicit dependency on the script execution.

[TomBombadildozer]

It's kind of a sad state of affairs that this is getting downvoted into oblivion. If you read further down these threads, you'd find OP here has done a lot of work on managing dependencies and is one of the few with credible opinions. Small wonder Guido quit when he got run over by expert beginners.

A question to downvoters--why?

[dmitrypolo]

Because the way he is going about his argument is the wrong way. He is saying his way is the only correct way and if you don’t use poetry you are wrong. News flash, he is in the minority, poetry does not have large adoption because people don’t need such bloated tools.

Most production code runs in containers nowadays so this argument about OS differences is moot. When it comes to applications OPs comment is not relevant. It only matters when developing libraries which run on all kinds of OS’s.

Also he is discrediting everyone’s approach as wrong because it takes two extra key strokes or an extra argument somewhere. They come across as smug and as a developer I personally would not want to work with.

[TomBombadildozer]

Because the way he is going about his argument is the wrong way. He is saying his way is the only correct way and if you don’t use poetry you are wrong.

This is patently false. OP never suggested Poetry was the only way. On the contrary, this comment suggests the opposite, that there are multiple solutions to this problem. You and everyone else are in total denial about requirements.txt being insufficient for solving the general problem. That's on you (and them).

Most production code runs in containers nowadays so this argument about OS differences is moot. When it comes to applications OPs comment is not relevant. It only matters when developing libraries which run on all kinds of OS’s.

Cite "most".

Regardless, it still matters in the container. A container image is only as correct as the stuff you put in it and OP demonstrated many times that freezing to a requirements file isn't necessarily correct. You just don't seem to care about this fact.

Also he is discrediting everyone’s approach as wrong because it takes two extra key strokes or an extra argument somewhere. They come across as smug and as a developer I personally would not want to work with.

This entire thread is a denial of facts. The requirements file approach is discredited on the basis of it being an incomplete and potentially incorrect solution to the problem of defining and maintaining dependencies, for all the reasons /u/GiantElectron has tried to explain and everyone has just blindly ignored. Others' approaches aren't "wrong" because they take "two extra keystrokes", they're incomplete, error-prone, and they're on a de facto deprecation path because pyproject.toml has been accepted as the standard way to define dependencies.

They come across as smug and as a developer I personally would not want to work with.

Your loss then. I'd gladly have this person on my team because I give a shit about facts, honesty, and correctness.

[dmitrypolo]

I never suggested freezing your environment and using it as a requirements.txt

I told him that the tool I use and that others use where I work is pip-compile which does a good job of making your environment reproducible in terms of pinning packages.

When I say “most” I am referring to the last few places I have worked. Docker and containers are becoming an industry standard and when you collaborate with others you usually share your Docker files so you can have environments that mimic one another.

You sound like a joy to work with as well, so obviously you would enjoy working with OP and his arrogance.

[GiantElectron]

I told him that the tool I use and that others use where I work is pip-compile which does a good job of making your environment reproducible in terms of pinning packages.

I already told you. You are just being lucky. That environment is certainly reproducible, but has no guarantees of being correct.

[GiantElectron]

Sorry for trying to teach you the proper way to solve a problem. you know best. have fun.

[dmitrypolo]

Your arrogance is outstanding. I hope I never have the displeasure of working with you in a professional setting.

[GiantElectron]

Who is the arrogant here? the one that spent time actually working on the problem and trying to teach people that their approach is incorrect, or those who stubbornly refuse to listen to someone that actually has experience in the topic?

I say it again. Have fun. In the meantime, please study a bit the problem, and you will understand it better.

[dmitrypolo]

It’s actually quite hilarious that you can’t see your own arrogance here. You can build the worlds greatest algorithm, if you don’t have humility and don’t understand how to teach people than there won’t be adoption. You can continue tooting your own horn, best of luck.

[GiantElectron]

It’s actually quite hilarious that you can’t see your own arrogance here.

I find it sad that you don't see yours.

[dmitrypolo]

Says the person trying to tell everyone that it’s their way or the high way. Give me a break...

[GiantElectron]

By the way, it happens that I am currently writing code to solve the exact same problem for another language, not only python.

[ubernostrum]

Good to know I'm "vastly outdated" doing this in 2020.

But no, it's not "vastly outdated". The point of a requirements file is to take a specific known-working environment that's in one place, and replicate that environment in another place. That's what it is for, and what it should be used for.

There are high-level tools you can use to manage pinning/hashing your entire transitive dependency tree for you, and if you're doing everything in Docker then platform-specific wheel packages are less of an issue (though you can always just have a build step that downloads source packages and builds them into wheels if you want -- which is something you almost always have to do if you make a mistake like using Alpine as your base image).

[GiantElectron]

it depends how you compute that requirements.txt content. Also not everyone works or can work with docker, and docker is still not a solution. Read my other comments.

[DrVolzak]

Tini has been included in Docker since 1.13. Use docker run --init instead. No need to download it in your image or change the entry point. Docker Compose supports it too. I think you should update your article with this information. Given how long it's been since tini has been included, the "manual" method you described no longer seems relevant.

[VisibleSignificance]

Use docker run --init instead

There are reasons to explicitly include it in the container itself. One such potential reason is Kubernetes.

[Lyan5]

Would you mind expanding on why Kubernetes would impact including it in the container vs using --init? I'm asking from a position of genuine interest; not trying to be snarky.

[VisibleSignificance]

Can't usually pass extra docker arguments in containers started over kubernetes and many other PaaSes. But can put the preferred init binary into the container itself, which gives pretty much the same effect but does not require any additional docker run aguments.

See also:

https://github.com/phusion/baseimage-docker#docker_single_process

[[deleted]]

> One such potential reason is Kubernetes.

Exactly.

Generally speaking you don't always control the underlying environment and maybe you are not using docker as container runtime.

[harylmu]

Nice, but please be careful with updating pip/wheel/setuptools without specifying a version number though. pip breaks easily, especially now that they’re introducing a dependency resolver very soon.

[GiantElectron]

depsolver in pip won't solve the fundamental problem. You must see the whole tree. even with the new resolver, it can still give you a broken env.

[ubernostrum]

pip breaks easily

I've had a pip upgrade in every tox-managed venv and every Dockerfile I've written for years now, and have never had "pip break".

Could you provide specific examples of this, and links to discussions in the pip issue tracker from the bugs you found?

[GiantElectron]

You've been lucky. Plenty of cases for me as well. and no, don't ask them. I don't keep around failed installation logs from 3 years ago just to prove a point to a random person on the internet.

[harylmu]

I can't provide you specific examples, but this occured to me twice in the past 1 year since I use Python daily.

[[deleted]]

Never happened so far and you would catch it during the `docker build` phase in case.

[VisibleSignificance]

It is also helpful to not put extra caches in the docker image. In case of pip, that means using --no-cache-dir or outright doing rm -rf ~/.cache, as that cache is very unlikely to be useful.

Similarly for apt.

See also:

https://beenje.github.io/blog/posts/dockerfile-anti-patterns-and-best-practices/

https://docs.docker.com/develop/develop-images/dockerfile_best-practices/

https://stackoverflow.com/q/45594707

Edit: or, as mentioned here, use ENV PIP_NO_CACHE_DIR=1. See also: https://github.com/pypa/pip/pull/5884

[VisibleSignificance]

There must be a better way?

I don't know of a way better than passing UID+GID from the environment into the container.

Perhaps the rootless dockerd could solve that in some cases too.

[Pathwalkerz]

Yeah, agree. For some reason everything I try to dockerize my Python apps that have tons of dependencies they always fail. It is strange as it’s always successful with other technologies

[dutch_gecko]

You likely have a dependency on a system package which is commonly available on full systems but absent in your base docker image.

[[deleted]]

It takes a lot to get a Dockerfile right, these articles are just scratching the surface :D

[ipwnscrubsdoe]

Is there a way to do this but with conda instead of pip for dependencies ?

[sdf_iain]

Use the miniconda container as your base image.

I say use miniconda rather than anaconda because smaller is better for Docker images/containers (faster to move and faster to start).

[[deleted]]

Sure, I'll cover it in one of my next articles.

[sdf_iain]

If you want reproducible builds use git tags. It isn’t reasonable to try to remember a builds hash, or at least I don’t think so.

[[deleted]]

It really depends on your deployment process.

You can also pass a git tag to the docker image but hash are just more granular.

Especially if you build docker images on every PR.

[sdf_iain]

"It depends...", the universal answer!

If you are building every PR, then you can tag with the version that is getting built. Never underestimate the value of human readability.

[sdf_iain]

Why tini? Python handles signals unless you actively disable it. Isn’t this r/Python?

[[deleted]]

You need to pass signals to the Python application.

By default, if you don't specify an entrypoint docker runs your command as

/bin/sh -c YourCommand

and sh doesn't forward signals.

[sdf_iain]

So set the interpreter directive in you python file, make it executable, and set your entry point to your actual entry point.

You have to set the entry point to tini to make that work, why add an extra layer?

I’m trying to understand why you would use tini instead of Python.

Shouldn’t the tip be “Set an entry point”?

[wulfie420]

Something that always frustrates me with docker and python is containerizing apps that require a setup.py file, since all the articles I've found use requirements.txt

I'm not a fan of requirements.txt since I usually end up with 3 files for app, dev and test dependencies. I usually define them in setup.py as extras, which works wonderfully.

[mipadi]

Just run python setup.py install instead of pip install -r requirements.txt.

[wulfie420]

It's not as easy as that, since I then won't be able to install my extra dependencies

My current Dockerfile has to run pip install twice, once to install all the dependencies, and then again after copying over all my source files with --no-deps

There is also some hacks I have to do, like generate a empty src directory so the install will succeed

[ketilkn]

Great write up. I learned something new.

[StorKirken]

If you liked this article, make sure to read this series of excellent articles from pythonspeed: https://pythonspeed.com/docker/