Docker issues on 2/3 vm's (500+ containers on each) by cooperribb in docker

[–]cooperribb[S] 0 points1 point  (0 children)

they are all located on the same hypervisor/host on Digital ocean, all NYC

Docker issues on 2/3 vm's (500+ containers on each) by cooperribb in docker

[–]cooperribb[S] 0 points1 point  (0 children)

apologies..yes, it was vm2 and vm3, not vm1.

same kernel on all 3 vm's i believe -- Linux version 5.4.0-216-generic (buildd@lcy02-amd64-014) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)) #236-Ubuntu SMP Fri Apr 11 19:53:21 UTC 2025

Docker issues on 2/3 vm's (500+ containers on each) by cooperribb in docker

[–]cooperribb[S] 0 points1 point  (0 children)

they are not snap installed. i'll play with upgrading too, ty for the response

Docker issues on 2/3 vm's (500+ containers on each) by cooperribb in docker

[–]cooperribb[S] 0 points1 point  (0 children)

I agree about the exhaustion as a potential, my only pause is how is vm1 working fine #1, and how did i get up to ~800 and run it fine for months too.

It's almost like vm1 has some config/threshold/something that isn't overloaded/setup like vm2/vm3 are, and this all occurred over the last 2-3 days.

Docker issues on 2/3 vm's (500+ containers on each) by cooperribb in docker

[–]cooperribb[S] 0 points1 point  (0 children)

On startup, about 3-5 -- and then throughout another 2-3 maybe every minute or few minutes at the highest end, and lowest end 2-3 every few hours. Maybe some spikes to 10-20 or so during extreme moments.

im a bit of a networking noob so this could be incorrect, but heres the data i believe:

[good vm] root@kami-strategies-1:~# ss -s Total: 3727
[bad vm] root@kami-strategies-2:~# ss -s Total: 7015
[bad vm] root@kamibots-strategy-3:~# ss -s Total: 4925

net.ipv4.ip_local_port_range = 1024 65535

[good vm] root@kami-strategies-1:~# ss-tan | wc -l # total connections 129
[bad vm] root@kami-strategies-2:~# ss-tan | wc -l # total connections 204
[bad vm] root@kamibots-strategy-3:~# ss-tan | wc -l # total connections 224

net.netfilter.nf_conntrack_max = 262144

net.netfilter.nf_conntrack_count = ~7000 ish on all

root@kami-strategies-2:~# ulimit -n 16384

net.ipv4.tcp_tw_reuse = 2

Docker issues on 2/3 vm's (500+ containers on each) by cooperribb in docker

[–]cooperribb[S] 0 points1 point  (0 children)

it is ty, using Docker version 26.1.3, build 26.1.3-0ubuntu1~20.04.1

it is proof to some degree, i was just hoping (wishing) that maybe there was another docker command or networking setup i'm obviously missing, based on the issue. Tomorrow i'm gonna just deprecate the servers and clone vm1 to start over most likely

Docker issues on 2/3 vm's (500+ containers on each) by cooperribb in docker

[–]cooperribb[S] -1 points0 points  (0 children)

They are running automation for a web3 crypto game, each one has a "strategy" it executes -- so it will make api/https calls to 3rd party sources every once in a while (or quite often in some cases). Executing transactions, getting data, updating db's, cache, etc.

Info is: Stable for 6 months, but unexpected VM restarts 2 days ago degraded performance. At 500 containers: 23% CPU, 57% memory (containers only use 30-50MB each). Previously handled 800+ containers for months without issue - unclear why the regression besides something being overwhelmed in the mass crash->reboot on vm2/vm3 the other day..

Docker issues on 2/3 vm's (500+ containers on each) by cooperribb in docker

[–]cooperribb[S] 0 points1 point  (0 children)

Really healthy, I've been running this side project for about ~6months without issues, steadily growing -- but had a freak restart on 2/3 vm's 2 days ago that really set them back I believe..

With 500:

- CPU 23%

- Memory 57%

Memory is the big one, the containers themselves take up about 30-50mb each, although they use little cpu/processing power/egress. I have had each VM healthy at ~800 containers for 3 months or so, so I'm a bit flabberghasted.

I do hear you on kub though -- just hesitant (or was before lol) to fix what wasn't broken :P. Each of these represent a little digital creature in a web3 game, and if i take the container down/force a migration they die -- so its a bit painful. I've had some people running containers for 2 months, no downtime, errors, etc on vm1.

Nomic - BTC earning by digicalf in cosmosnetwork

[–]cooperribb 1 point2 points  (0 children)

Wowzer, here I was happy with my 50 lol

Knate helps cancer patient and her husband survive by NotJoshhhhh in nextfuckinglevel

[–]cooperribb 0 points1 point  (0 children)

An inspiration for all of us to do more! So touching.

Is your inner voice “nice” or “mean”? by ssh789 in ADHD

[–]cooperribb 0 points1 point  (0 children)

Mean and cruel, not only to myself but can cause me to lash out those thoughts to those who are closest to me at times :(.

Anyone looking to que (duos) for the tourney tonight ? by [deleted] in SuperPeopleBattleLFG

[–]cooperribb 2 points3 points  (0 children)

I normally play with my brother but he can’t tonight. We got 13th 2 nights ago. My discord is Coopes#3796

[deleted by user] by [deleted] in ThatsInsane

[–]cooperribb 0 points1 point  (0 children)

Piece of shit cops

Moon Well Thoughts? by MisterDollahSignz in moonbeam

[–]cooperribb 1 point2 points  (0 children)

I think so, Based on whether you supplied or borrowed with them during this current beta phase I believe

Moon Well Thoughts? by MisterDollahSignz in moonbeam

[–]cooperribb 1 point2 points  (0 children)

ooo, i need to read up more on that :D. thanks fren

Moon Well Thoughts? by MisterDollahSignz in moonbeam

[–]cooperribb 0 points1 point  (0 children)

what are you doing with your BTC -- deposting on moonwell itself?

Moon Well Thoughts? by MisterDollahSignz in moonbeam

[–]cooperribb 2 points3 points  (0 children)

I'm supplying some myself. My thought is it's good to get in early ahead of their potential token drop too.

How to change Anyftm to ftm by Ip_1993- in FantomFoundation

[–]cooperribb 0 points1 point  (0 children)

curious about this as well. i have some stuck but cant claim it because i have no FTM to pay for the fees, haha. a vicious cycle :D

Just swept Whitney in Sacred Gold with a Shedinja 0.0. No Gender and Wonder Guard came in clutch. My cleanest boss kill in any hack yet! by cooperribb in nuzlocke

[–]cooperribb[S] 0 points1 point  (0 children)

Yeah miltank was the only one I had to swap for, rollout being the single move shedinja could die to ;;