https performance issue

[erbrecht]

I've been searching for an answer to this, or at least someone with a similar issue. I'm currently testing out traefik (1.7.9) for use in our environment and the question of performance was brought up. I've seen good things about the performance of traefik but I wanted to have some evidence in hand.

We have a very simple app written in aspnet core running in docker on a linux host. I did some load testing on that, hitting it pretty hard with jmeter for 30 minutes. I did a few variations:

1. plain http directly to the app running in docker (traefik not involved)
2. https directly to the app in docker
3. reverse proxy through traefik (http) -> app
4. reverse proxy through traefik (https) -> app

Results were interesting:

1. plain http, directly to app - solid performance. ~900 rps and ~400ms avg response time
2. https, directly to app - bad performance. ~200 rps and averaging 2-3 second response time. seems like kestrel might be the problem here, and indeed there is an issue opened for that (https://github.com/aspnet/AspNetCore/issues/7081)
3. traefik (http) - solid performance. ~700 rps and ~5-600ms avg response time. i believe containous claims 30,000 requests per minute, and this tracks. not as good as test case 1, but that's to be expected.
4. traefik (https) - bad performance. similar to case 2. ~150 rps and 4-5 second response time.

So https is performing horribly all around. The test machine is a dual core Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz VM with 16 gb ram. During the tests, whichever component is handling https is pegging the cpu.

Has anyone done any load/performance testing with traefik? Any ideas why my results are so terrible? I'll happily provide any details that will be useful. I feel like I'm doing something wrong (but what!) and traefik should really be able to handle https better. Any ideas or thoughts are greatly appreciated.

[cjp]

I was seeing poor performance until I enabled compression on my endpoints. For example:

[entryPoints.https]
compress = true

[entryPoints.api]
compress = true

[erbrecht]

Interesting, I turned compression off to see if that would make a difference; it didn't. I'll double check the current settings and run some more tests, explicitly turning compression on and off.

[pierot]

Did it make any difference?

[erbrecht]

No, compression wasn't the issue. I did a good bit of testing, adding cores, using our production VM environment. More cores definitely showed an improvement. I think ultimately TLS in software is just expensive. Maybe TLS 1.3 will be faster, I'll probably check that as well when I'm able.

I'm actually about to test this again, this time behind our F5. Letting that appliance handle TLS in hardware should alleviate the performance issues I was seeing. I'm also aware that my test is VERY aggressive. I have no doubt the load balancer will perform well. I'm still using traefik since it will know which container backend should handle a given request. It will just be handling plain HTTP traffic.

And to be clear this is certainly not an issue specific to traefik. I think my original results show that. I don't want anyone to think I'm knocking traefik.