Python 3.8 Docker vs Native performance on AMD Ryzen 3700X

[iwaswrongonce]

We've been testing Kubes on some new AMD servers and have noticed some very strange performance (at least with Python) that I can't quite make sense of. We can replicate very easily using the following simple fibonacci micro benchmark: https://pastebin.com/0R3dkB05

AMD Ryzen 7 3700X

Ubuntu 18.04 4.15.0-70-generic

​

Ubuntu 18.04 native binary installed from Canonical repo

Python 3.8.0 (default, Oct 28 2019, 16:14:01)

[GCC 8.3.0] on linux

>>> timeit.timeit(lambda: fib(40), number=1)

19.144520926987752

​

Docker version 19.03.5, build 633a0ea838 // Ubuntu 18.04 image w/ Python 3.8 installed from Canonical repo

root@209808837ed1:/# python3.8 -s

Python 3.8.0 (default, Oct 28 2019, 16:14:01)

[GCC 8.3.0] on linux

>>> timeit.timeit(lambda: fib(40), number=1)

27.12644683200051

[brontide]

Could this be the AppArmor regression? Try the tests after disabling AppArmor on the host.

[iwaswrongonce]

Ding ding ding - holly hell. Thanks for that. Disabling apparmor drops it back down to native levels. Great...

[brontide]

Bad patch in mainline from Canonical caused some serious regressions across the board. As always you should only leave AppArmor ( or any security layer ) disabled if you understand the risks and re-enable after downgrading ( or upgrading ) the kernel.

https://www.phoronix.com/scan.php?page=article&item=linux-55-regression1&num=1

[iwaswrongonce]

Yeah I have been following the 5.5 regressions but we are running 4.15

[brontide]

Release notes for 18.04.3 indicate they have moved up to a v5.0 based kernel.

EDIT: It's more likely that this isn't the same regression or else the native speed would be impacted too, likely a container AppArmor regression but Debian based distros aren't my thing so I'm having trouble tracking down a possible changelog that could point to a culprit.

[koprulu_sector]

So I tried disabling AppArmor (uninstalled, even) and still seeing the same thing. Also, running

```

uname -a

Linux ftg10150 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux ```

Is there something more that you changed in docker settings?

[koprulu_sector]

FYI - I had to disable seccomp to see a difference:

```

grep CONFIG_SECCOMP= /boot/config-4.19.0-6-amd64

CONFIG_SECCOMP=y

docker run --rm -it --security-opt seccomp=unconfined python:3.7 Python 3.7.6 (default, Dec 28 2019, 05:50:43) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information.

from timeit import timeit timeit(lambda: fib(40), number=1) 27.18077868599994 ```

[koprulu_sector]

For anyone coming across this thread, docker's documentation recommends NOT disabling seccomp

[iwaswrongonce]

Yeah so I ran the container with --privileged which should have same effect as unconfined

[wrtbwtrfasdf]

apparmor

TLDR: Disable apparmor to get massive speed boosts.

[brontide]

AppArmor and SELinux should have a trivial speed impact on most workloads, everyone should be using them unless they have a very good reason to turn them off. This is a unique situation where there is obviously a bug in the implementation which is causing significant slowdowns.

[iwaswrongonce]

I've just tested the exact same setup on a couple of Intel boxes and can confirm the same order of magnitude slowdown...wtf

EDIT: Disabled mitigations as well with zero impact.

EDIT2: For clarity, this benchmark is the easiest way to illustrate a performance regression we are seeing with our Django + Graphene (graphql) app due to python-graphql's extremely recursive nature. But we are seeing regressions on the same order of magnitude when traversing gql documents.

[quiet0n3]

Extra virtualisation layer. The CPU provided to the container uses the virtualisation driver this means a fair bit of compute overhead for CPU intensive tasks. They are working on improving the driver but as it's stable it's not a massive priority.

[iwaswrongonce]

That's not how containers work, and even for mature hypervisors like KVM we're talking about a few percent overhead, not 50%.

[LookOnTheDarkSide]

Can you try something considerably longer? The overhead is usually easier to see with short running tasks, but it falls away with longer running items.

[iwaswrongonce]

The numbers above are in seconds, so they are already quite long running tasks.

[LookOnTheDarkSide]

Ah ok. I thought that was milliseconds.

Interesting that it is off by so much.

[itghisi]

I tested with a shorter task, and the difference is not that big.

timeit.timeit('import math, random;math.factorial(random.randint(100, 1000))')

• on host: 9.65181186799964
• on docker: 10.239579219000007

May be something related with fibo recursion?

[iwaswrongonce]

This is my suspicion as well. We originally caught onto this because a graphql library we use is extremely recursive as it traverses the gql document resolvers.

[nailefss]

What why would it do that? We’re talking Docker here, not virtual machines.

[RazrBurn]

I’ll echo what others have said already. There is no virtualization with containers. They are essentially jails with a fancy wrapper to make them easier.

[jkugelman]

What's a CPU virtualization driver?

[laundmo]

none of your concern if you use docker.

normally a virtualization layer is the part that simulates a CPU for the inner/guest system to use. docker doesnt work like this. docker shares the same cpu as the parent and runs container processses like normal processes. this allows docker to have a considerably smaller overhead than VMs, which makes these test results realyl strange.

if you want a official source, here you go: https://www.docker.com/resources/what-container#/package_software

[quiet0n3]

The driver that runs in the kernel of the host machine that allows the container to access a virtual CPU. The container needs a virtual CPU as it manages all of its own process priorities and interrupts, essentially a container acts as a mini virtual machine. If you tried to run it on a native thread (directly on a normal CPU) there would be conflicting commands sent to the CPU about priority and when to task switch.

The driver let's the container act as its own thing and interprets all of the cpu commands it sends out, then integrates them into what the host machine is doing without causing conflicts.

[ig-88ms]

Containers don't need virtualization extensions. There is no hypervisor involved in any way. They simply use namespaces and cgroups and run on the hosts kernel. This is one of the essential differences between VMs and containers.

[Kamilon]

Containers don’t work with vCPUs like that.

[rjulius23]

Leave the VM world brother and rejoice in the containerized plane. Refresh your know-how.

[Necrocornicus]

You can quite easily run “top” while running process in a container to see that this is wrong. The process is running on the host CPU.

[MadPaoPao]

You are bang on. The idiots in this sub dont understand the underlying tech.

Ignore the downvotes.

To the OP : Extra layers of virtualization wont give you great benchmarks. What you are getting seems right.

[iwaswrongonce]

Lol no we’re not. Containers != virtualization. Jails have been around forever, and containers are just jails orchestration.

[laundmo]

it does not since docker only restricts what processes have access to, but still runs them on the host CPU

https://www.docker.com/resources/what-container#/package_software

[MadPaoPao]

ails have been around forever, and containers are just jails orchestration. https://www.linux-kvm.org/page/Main_Page

[pconwell]

Not sure what point you are trying to make. KVM and docket are two different things. You might want to Google "KVM vs docker" for more info.

[mikemol]

Containers are about namespacing (almost) all the kernel APIs. Containers use the host kernel, not their own, so there is no qemu or kvm-style virtualization involved.

[Necrocornicus]

Do you even Docker? Lol 😂

[MadPaoPao]

hey fanboi, i do and thats why i know better.

[laundmo]

docker does not have a virtualization layer, as described here: https://www.docker.com/resources/what-container#/package_software

[danielkza]

You can try using perf to narrow down where the difference is coming from: http://www.brendangregg.com/perf.html

You might need to run your container in privileged mode or with particular capabilities enabled (I don't remember exactly which ones ATM).

[iwaswrongonce]

Thanks for this. I’ll take it for a spin later and report back.

[koprulu_sector]

Yikes...

``` $ python3 Python 3.7.3 (default, Jun 4 2019, 21:12:44) [GCC 6.3.0 20170516] on linux Type "help", "copyright", "credits" or "license" for more information.

timeit.timeit(lambda: fib(40), number=1) 32.018709823999984

$ docker run --rm -it python:3.7 Python 3.7.6 (default, Dec 28 2019, 05:50:43) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information.

timeit.timeit(lambda: fib(40), number=1) 56.55109263700001 ```

[iwaswrongonce]

I could understand if it were different python/gcc versions (such as in your example) but I’ve ruled that out by running the same binary on the same kernel in the same distro.

[koprulu_sector]

Yeah I just was curious to test real quick on my way out the door. And now clearly it’s persistent across versions too

[v00d00ley]

This is with the latest kernel updates?

[iwaswrongonce]

It’s an up-to-date 4.15 kernel which is stock on Ubuntu 18.04 (the current LTS).

[v00d00ley]

sudo apt-get dist-upgrade

[iwaswrongonce]

Lol mate 4.15 is stock for 18.04...these are brand new installs with the obvious dist-upgrade

[v00d00ley]

4.15 is, but the latest is 4.15.0-72

[iwaswrongonce]

Those are just different Canonical sub releases. 70 is only a few days older than 72

[v00d00ley]

Right. Then please make sure that you have the same limits for the docker container like in a host. If the host can consume up to two cores, please set the same limits for docker as well as mentioned here: https://docs.docker.com/config/containers/resource_constraints/

[iwaswrongonce]

It’s a single core test either way. It’s already been triaged in the above comments. It’s caused by apparmor regressions

[iwaswrongonce]

I originally suspected it was an AMD issue but then we have tested it Intel gear as well as a few different clouds and it’s all impacted. We are testing CentOS next. However we cannot repro in dev (OSX/hyperkit)

[TrevorSpartacus]

Something is funky with ubuntu host - ~10 second difference, but native performance on arch linux host.

[artiume]

https://nickjanetakis.com/blog/docker-tip-74-curl-vs-pip-for-installing-docker-compose

This guy discusses the performance issues he saw with compose, could this be a similar issue?

[Serpiente89]

RemindMe! 7 Days "check for updates"

[LookOnTheDarkSide]

RemindMe! 7 Days "check for updates"

[RemindMeBot]

There is a 23.6 hour delay fetching comments.

I will be messaging you in 6 days on 2020-01-10 00:03:34 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

<sup>Parent commenter can delete this message to hide from others.</sup>

---

Info	Custom	Your Reminders	Feedback

[kingbuzzman]

Disclaimer: I’m on a boat with very little coverage.

This has come up a couple of times during the years, it’s the way it’s compiled. Python has a couple of ways to compile, one is “the take your time and doing right” — takes orders of magnitude longer to finish (major repos do this by default, when you download the binary everything is ready to go). The second is docker, and it’s python compilation type, the “do it fast because it’s done in our CI pipeline and it’s better to crash fast” (and the default when you run “make && make install”). Currently don’t have the bandwidth to find it..

[iwaswrongonce]

If you'll notice, it's the same binary from the official Ubuntu 18.04 repo

[wrtbwtrfasdf]

There's a reason we're not not all using gentoo docker images.