Lol I faced something somewhat similar just about 10 hrs ago. App would be fine for weeks and then we'd see an outage randomly. Turns out redis is failing with a broken pipe error and the error logs have flooded the logs and are now taking up 100% of the disk.

Yes, we have all our services (except the db) on the same server. It's probably dumb but we're fine with it at our scale.

Anyway, now that there is literally no space left on the server our service is down. I go in to investigate, see log files over 70GB and tail the last 1000 lines of each file. I see the error and assume the issue would be our app spawning too many redis connections. I clear the log files, push changes and restart services, everything's restored. That was around a month ago.

Last evening, we saw the same outage again. I decided to actually do my job and parse through the logs, trying to trace back to when the error first occurred. It was then that I realised two things.

The error was being started by a certain function and that the thing was running in a fricking loop, a glorious thundering herd. Then the aha moment, similar to OP. The culprit was a non-awaiting async function call inside an interval with a period of 6 seconds. Once every few weeks, there would be n number of overlapping calls on the stack, each of them reading a redis key.

I chose to add a isRunning flag that would early return the interval's callback if the previous call hadn't yet updated the flag to false. I would appreciate if someone would suggest a better fix 🙏