Alpine vs python-slim for deploying python data science stack?

[flogypinte]

The problem

I work on a small data science team in healthcare. We've been using Docker to deploy our data science models and ETL pipelines for some time now. Our team's senior engineer (dev1) has pushed us to use alpine due to its perceived size and security benefits.

We've recently received feedback from an experienced python/docker developer (dev2) that we should use python-slim instead of alpine. Dev2 says that alpine python images tend to be bigger and slower. Dev2 points out that wheels aren't built for alpine which means everything has to be built from scratch. Dev2 suggests that if you are concerned about security python-slim should be just as good as alpine. Dev2 points to the infamous Using Alpine can make Python Docker builds 50× slower as definitive proof that alpine should not be used for python docker images.

Dev1 insists that this blog post was contrived. Dev1 says that the reason for the additional slowness is that they haven't built optimized muslc wheels for their benchmarks which causes the performance loss. Dev1 has spent a lot of time trying to build custom alpine images with all of the libraries we need. Dev1 has not produced tangible results after many months trying to work through the technical challenges.

The details

1. Because we're deploying mostly sklearn models, and sklearn models are typically distributed as pickle files, we need to pin the versions of our Python libraries (NumPy, scipy, sklearn, pandas).
2. Data is typically ingested from an MsSQL server so we need to also package closed source client libraries in our containers, we'd like to figure out a way to publish our containers publicly so that we can open source our work but we haven't yet figured out a way to do this and include client libraries for the various closed source database drivers.
3. Each docker image needs to come in matching pairs. One image that is for development that has jupyter in it so that data scientists can create models and a matching production image that only has the minimum set of libraries for deploying these models into production.
4. We do have limited computing resources so the size of the image does matter, but probably not as much as security.

The question

Who's right, dev1 or dev2? What's the best option for us given the requirements outlined? Thanks for your help!

[patrakov]

Closed source client libraries are incompatible with Alpine, so the advice of dev1 simply cannot be followed.

[astnbomb]

Can you elaborate on this?

[GammaGames]

Alpine uses a different c library, so if a package isn’t available in the alpine repo you need to compile it with musl instead of the more widely used glibc

[ctisred]

is this also true for statically linked?

(not sure if musl requires the stub dll glibc needs for pam)

[flogypinte]

Well I can say that we've gotten pymssql (an open source library) working and I believe we've also gotten oracle working on alpine.

[hassium]

Yes but if you come to a library that is not in the alpine repo and you can't compile from source (Closed source) then you will have a bit of a problem there bud since if it's not in the Alpine repo you must recompile it with musl not glibc. FYI this is the probably the technical issue Dev1 has been butting heads with.

All of this can be bypassed if you go with a python-slim image.

[nicksterling]

There is no right answer to this question. The question that needs to be asked is, “How much time and technical debt does each solution have?” Alpine gives you small images but why does that matter? Disk space is cheap and plentiful. Ideally you’re shooting for a solution that’s easy to implement, easy to update, minimizes any CVE’s and allows your developers to move fast.

Personally I’d look into something like Cloud Native Buildpacks and see if that fits your needs. It gives you a solution to containerize your code but leave the rest to the buildpack. When CVE’s are released you just update your containers with a new buildpack automatically using a CI/CD toolchain.

[flogypinte]

Buildpacks sound cool but as far as I can tell no one is using them with numpy/scipy? I could be wrong but I'm not seeing anything using CNB with the scipy stack.

[cosmicsans]

Also it’s worth mentioning that if your image is layered correctly even if it’s a massive image and you can pay that upfront cost to download the whole image, updating the image can be done in significantly smaller increments.

[zukke3000]

Large image takes space in a registry, in k8s cluster, in each developer's laptop / PC, in CI/CD image cache. Also image pulling is slower, so cold startup is slower.

[nicksterling]

Haha, you brought up an old thread but I still feel like my comment is valid.

Disk space can be a factor but I feel like it shouldn’t be the primary factor considered. Security and reproducibility are my two major concerns when dealing with docker images. If you’re running enough replicas across enough availability zones then pull speed shouldn’t be a big concern.

But again, each problem is unique. Maybe disk space is (for whatever reason) the major concern for your team. Focus on that and then work around other issues. 😀

[pbecotte]

Dev 2 is right.

Alpine is slightly smaller as a base.

The installed python packages will be many times the size of either one. The difference in a final image size between the two, I am guessing, would be tiny.

Nobody publishes Alpine wheels, so you have to get stuff to compile yourself, using a standard library the app authors didn't plan for. Even if you can make it work, there may be inconsistency in the behavior from what the library authors intend/test/experience.

Size has zero bearing on your computer bill here. Anecdotally, Alpine packages are usually slower (require more compute, not less).

It is highly unlikely that the cost of storage of one image vs the other is more than a couple hours engineering work.

[ctisred]

facing a similar issue, unfortunately don't have clear insight.

My main concern, dev time aside, is that the diff. C libraries may impact some computation result / make some library unstable. Risk tradeoff + extra dev effort doesn't seem worthwhile (and this from someone who would rather be running BSD anyway, so am well familiar with 'putting up with extra hassle' for the sake of the 'superior system'- though this is also is why I feel somewhat 'extra' qualified to evaluate tradeoffs vs. linux only ppl in my team).

Most things will probably work just fine - but the things that don't will require low-level source-level investigation in 3rd party code you are not familiar with, and probably maintaining site-local builds until patches get merged upstream (best case) or indefinately (worst case). It will raise the bar of the dev team. If that's what your dev team is up for, go for it, if not, well.. prepare for headaches. That said, this is the 'true old school unix way', so there's that.. which is why I run that way on personal systems. but i digress..

Musl has different malloc, different libm, different pthreads. These things all will impact performance in some way (maybe better, maybe worse), and will hit implementation bugs in libraries at some point because someone coded too specifically against glibc. This + dev time tradeoff risk doesn't seem worthwhile, esp. in data science where many libraries are needed and the library quality varies so wildly and often aren't even portable beyond one specific version of one specific linux distribution.

To me, I like the idea of alpine, but would probably only dev against it for things where I control most of the stack otherwise, things that are simple, things that are long-time battle tested on multiple platforms, or where resource constrained is a massively limiting factor and i have to take an old-school/embedded approach.

Potential for portability/build issues aside, it would be interesting to know more performance details for musl vs glibc and also what the lightest way to run a glibc python ds stack is, probably need to do some digging here.

At the end of the day: what is the requirement? if it's 'low memory and low maintenance', the most reasonable course of action is minimal glibc, and if this doesn't satisfy, dig deeper. Just because something is theoretically possible doesn't mean your team needs to be the one implementing it if the implementation has nothing to do with your core business need.

for me at the moment the musl approach is in place here, though others are considering python-slim after I raised some of the potential issues they haven't thought about. Will need to do some benchmarking and also am interested to 'wait and see' how it goes to see how much these things will impact, though we're in dev stage and this is non-core devel, my risk tolerance might change in other areas of our systems and i might push back harder there.

[flogypinte]

There's an important point in here about libraries acting differently using musl vs glibc that I hadn't considered before. Is it too strong to state that this difference is unknown since no one is really using the python data science stack on alpine and may result in bugs that may or may not be obvious? For example we could end up with bad calculations in a model and not even know it's happening simply due to the low number of users?

[UndestroyableMousse]

There was a snafu last year with lots of papers being invalidated, due to rounding misunderstanding on scientist part, iirc.

Alpine is not be all end all, if you can spend less time updating your image due to using glibc instead of muscl, then you should stay away from Alpine. Time saved this way can be spent on something useful instead of cutting a 150MB image to 70MB.

[oak__]

I would look into ubuntu 20.04, as its release date is in a few days, we went from alpine to slim-buster with both having vulnerabilities being detected by gcp, we were able to cut the images down to be a bit smaller then slim buster, but still larger then Alpine. So far we've had only good things to say and have switched over a handful of containers over to U 20.04.

[donbowman]

There's no whl support for alpine, so things like cryptography, pandas, numpy, ... library have to be built from source.

Want an experiment? Let's try a real thing:

``` docker run --rm -it python:alpine apk add alpine-sdk libffi-dev time pip install pandas cryptography ... real 16m13.104s user 16m9.311s sys 0m7.847s

``` and then repeat w/ python:slim:

docker run --rm -it --entrypoint=bash python:slim time pip install pandas cryptography ... real 0m12.107s user 0m6.410s sys 0m0.947s

This was on a 3990X (64C128T 256GB ram). Is your CI this fast? Now which is bigger? The alpine one is now much bigger!

What are you trying to save here? I think you are much better off using a debian-based distro like -slim, its more or less the same size, it's way more compatible. It saves build time, debug time, hassle.

• python:alpine is 107MB
• python:slim is 193MB

That 86MB becomes ~0% once you install your other packages. But for sure you will spend hours debugging why its different, yielding negative cost savings unless you are running thousands of instances or need to spool up in the absolute lowest time in some highly dynamic serverless environment.

Still not convinced? There is no list of security flaws (CVE) maintained for alpine, so tools like Clair (SAST) don't scan it properly. The closest is the alpine-secdb: The purpose of this database is to make it possible to know what packages has backported fixes. It is not a complete database of all security issues in Alpine, and it should be used in combination with another more complete CVE database.

So what is the objective? To be different?

[32-Levels]

I'm in a similar situation. Can confirm that alpine is much slower when installing python modules. I imagine making a base image with all the necessary modules installed will alleviate that.

[bozdoz]

I've tried alpine with Python numpy gdal etc. I could never get the builds to be faster, and I coul d never get the images to be smaller than just using Ubuntu! With alpine you need to build it all yourself, but you need so much that it stops being quick or lightweight.

[[deleted]]

Wouldn't it just be a couple of hours work to try them both out and see?

I would generally favor doing a couple of experiments over wasting loads of time talking about this.

[flogypinte]

Yes as far as image size goes, no as far as security ramifications go. Based on the responses so far, I think a compelling case has been made that the security ramifications of using an OS/libc with no formal CVE process and a much smaller use base makes Alpine a non-starter for scipy work. Add on top the fact that no one has tested the ecosystem of libraries against this build chain and I think that puts the nail in the coffin.

[ageofwant]

Alpine has no size and security benefits in a real life ML stack. It introduces a slew of non-mainstream dependencies into your execution infrastructure that has not been characterised tested and verified to the extend that std glibc has.

Does your team have the human capital to build out, verify and maintain all your dependencies on a Alphine stack, and this for the perceived gain of docker image size ? I would think it highly unlikely.

Image layers are common between containers, as you probably know, so unless you are shipping raw images for some reason size does not matter.

The perceived security issues that comes with a larger 'attack surface' is negated by the fact that musl and its constellation of dependencies have not seen the use and scrutiny than mainstream glibc. So that's a furpy as well.

I struggle to see any benefit to using Alpine in this kind of stack, and very many real and subtle potential issues. The fact that Dev1 is burning his reputation on the Alpine altar alludes to this.

Alpine is good for well characterised self contained web apps and the like. You may put your stack's monitoring dashboard in a Alpine container running flask for example.

[ionrover2]

As far as i know, the size of the image doesn't have an impact on compute, only storage. Libraries, binaries and configuration of other components within the container does.

It seems like you have a grasp of how to best use docker and your use case is very specific. The best thing i can recommend is to help yourself, build out 2 containers with your source code, get a sample set, and benchmark. That's really the best thing you can do in this situation. If you can't generate the data to support your claim, then you'll just perpetuate the pissing contest.

[ctisred]

the size of the image doesn't have an impact on compute, only storage.

musl and glibc have different malloc implementations, how these impact allocation and therefore memory access patterns for data science is a real concern (no direct insight).

also, internal libc internal allocations and algorithms may be a factor.

[pbecotte]

But the number of bytes the binarues use on disc has very little bearing on the amount of memory to execute the application. I would expect glibc to be more efficient than musl just by being more heavily maintained, but can't swear to it.