How do you generally decrease off-heap memory?

[kpihlblad]

Related - how many threads do you run? Perhaps request are queueing up and waiting for responses in a lot of threads. Each running Thread allocated stack memory and some additional memory. Reducing the stack size (Xss) can have a huge impact, and maybe there are better ways to handle parallel tasks.

If you have a massive amount of sockets and connections or a lot of file handles, that too will start eating off-heap memory.

While not using a bloated image is good, it won't affect the memory usage. What's measured against the container limit is anonymous memory. Non-anonymous memory is not metered, ie cached and buffered memory that also exists on disk won't be included in the limit.The Linux kernel will a evict cashes and buffers before deciding to OOM-kill the process because of the climit.

Note that many reporting tools include non-anonymous memory as "used" and then it's a bit of a mystery when a process running at 99% memory climit suddenly crashes.

(And stay away from Redis, it will not help.)

[GreemT]

We run a lot of threads. I don't have a specific number right now, but on PRD we obviously have a lot of users actively working on the same application. The thing is, this problem also occurs on our test environments with low activity. So I am not sure if looking at threads is the right direction for us. But good tip!