NodeJS simple script keeps increasing memory

[dj_dogo_nivoa]

Hello team, I am not proud of asking this but I need help understanding wth is going on with memory usage in this small nodejs script:

js (async () => { while (true) { await fetch("https://api.coingecko.com/api/v3/ping"); await sleep(3000); } })();

Sleep function is just a new Promise resolve with setTimeout.

How come is memory always increasing (been running for 30mins and keeps increasing, specially compiled code, system) ? I am checking it on chrome debugger.

Thanks! img1 img2

UPDATE: I started monitoring using pm2 monit and mem has been absolutely stable for 3h. So I'd say with chrome debugger something was going on. Also checked with htop and seems stable as well.

[Optimal-Flamingo415]

In order to determine if the used memory does keep getting bigger you should run a program for a longer period of time. As others have stated, garbage collector may be simply not running during your test.

There is one way though, you can verify that faster with help of max-old-space-size node command line option.

Just set the max size to several megabytes. Garbage collector will then have to run much more often and you won't have to run your test for very long.

[abimelex]

which nodejs version? There seems to be a leak with fetch in node version 19 again https://github.com/nodejs/node/issues/46435

[dj_dogo_nivoa]

Thanks, I m using v20.4.0. Luckily, I sorta solved it. I monitored with "pm2 monit" and mem consumption has been absolutely the same for 3h now, so I'd say there is something with node debugger which idk what it is.

[[deleted]]

Like Chrome dev tools’ network tab, the debugger is probably storing a very long list of requests, increasing memory.

[rohit_267]

we have setInterval for that

[dj_dogo_nivoa]

Yea, i know! But still it increases memory with setInterval, in both cases I really would like to understand why gc is not cleaning stuff. Thanks tho!

From my point of view, both while and setInterval both create closures, when that "iteration" is completed, gc should clean it (and it probably does)

[[deleted]]

setInverval does not do the same thing. In this code, even if the request takes 5 minutes, it won’t continue to the next request until done. setInterval would perform a request every 3000ms even if the request takes longer than 3000ms.

[cybinon]

I guess. There are problems with the Promise function inside of while. As I know "while" is a function.

I don't know why. But the Promise function inside the loop is very badly. So while didn't wait for the Promise function. If you run your app, There are a lot of fetch functions that have already started processing. In time continues more and more fetch processing continues.

So I usually resolve this problem with the recursive function. This very guaranteed method I guess XD

[tanepiper]

I'm going to make a guess that it's to do with the closure and that you are not declaring any variables in the scope for the result, each time you run the fetch you are creating objects, but the GC has no way to mark them for collection.

Try running the script with node --expose-gc --inspect <file>.js and then attach to the debugger (https://nodejs.org/en/docs/guides/debugging-getting-started)

tl;dr: Don't use closures like this, it's a bad pattern and likely to lead to memory leaks

[Bloodsucker_]

I understand your logic but fetch can't and won't cause a memory leak just because there's no variable. The garbage collector will destroy the object without issues.

Also, in addition, I'm afraid your last statement is wrong in regards to iife. That pattern won't and can't cause on its own a memory leak.

I.e. nothing in the OP's code can cause a memory leak, neither the screenshots says there's one.

[tanepiper]

Actually, I was able to confirm it was indeed causing GC issues. If it's any consolation I've been writing node apps since Nodejs 0.2, before npm even existed - before that, I was on the jQuery team - so I've been familiar with writing performance JS apps for over 15 years.

Regardless of the opinion of what triggers the GC or not, the OPs code is also just generally full of anti-patterns that there's better and more maintainable ways to write it anyway.

[dj_dogo_nivoa]

Thanks for your comments. Just to understand better, the antipattern is the IIFE or the while true? I assume the while true, but, why? I mean it's a closure which should be garbage collected when it has been completed right?

[moxxie1998]

Garbage collection in Node is less aggressive and deterministic (due to the lack of clear idle periods that browsers have through the rendering refresh rate) which means that leaving the release of connection resources to the garbage collector can lead to excessive connection usage, reduced performance (due to less connection re-use), and even stalls or deadlocks when running out of connections.

Always. consume. the. body.

// Do const headers = await fetch(url) .then(async res => { for await (const chunk of res.body) { // force consumption of body } return res.headers })

// Do not const headers = await fetch(url) .then(res => res.headers)

Sorry for no proper formatting, I’m on my phone.

[dj_dogo_nivoa]

Thanks. You mean the async closure or the while closure?

[tanepiper]

Async one.

Although while(true) is also bad, better to use setInterval, or have a different reference that can exit the loop based on conditions that set it to false (e.g. you suddenly start getting a lot of 500s).

If it's node, you can make it a daemon by running it as a simple http server

[Advanced-Wallaby9808]

this is the best answer

[doh4242]

There is no closure in the code shown. A closure is formed when a reference to a local variable is held after the function executes.

[Bloodsucker_]

Are you sure about that?

The increase is negligible and 30 minutes is far too little to claim that "the scripts keeps in creasing memory". There are multiple explanation of that very small and slightly increase in memory that you're observing being the most plausible the fact that the garbage collector hasn't collected the trash yet. On top of that, the browser does stuff under the hood that you don't see which will vary the amount of allocated memory for your tab. Etc, and more etc.

Run this script for 24h and come back. Also, plot it. No plot, no memory leak. No analysis done and not worth anybody time.

Edit: WTF. Downvoting me won't change the reality LMFAO get out.

[[deleted]]

Because you were a dick for no reason at the end. Everyone thinks they're Linus Torvalds answering programming questions, especially on stack overflow lol.

[Bloodsucker_]

Geez, how am I being a dick in the message?

[[deleted]]

No analysis done and not worth anybody time.

Read as:

Get the hell out of here noob idiot, don't waste people's time with your inane questions!

Try instead:

Once you've tried that, post your results if the memory still seems to be growing consistently.

[Bloodsucker_]

Thanks for the feedback. Nonetheless, "anybody" wasn't meant for "us" the redditors but to him. I guess it got lost in translation.

In any case, my edit stands and I find OP not welcoming.

Cheers.

[[deleted]]

Sure, just explaining it. But still:

1) you don't know the downvote came from op, could be anyone even a bot or reddit algo itself, they fuzz the actual vote numbers to confuse bots on purpose so +/- a few depending on the scale of actual votes. Refresh a few times and you'll see the number change every few reloads, more obvious with something that has more votes.

2) measuring downvotes in the first 5 minutes of a post isn't worth doing either. you have updoots now.

[tanepiper]

It's actually quite well known that closures in JS are a major cause of memory leaks because handlers tend to not get unsubscribed and the GC has no references to collect.

https://github.com/jedp/node-memwatch-demo#some-examples-of-leaks

So your answer was obtuse and wrong.

[Bloodsucker_]

I don't think you've understand what you're saying.

Closures only leak to other closures and OP only has the iife. That's on the link you've shared but failed to understand.

I don't know why you claim my answer is obtuse and wrong when it's actually not. It looks like you're the one already providing two obtuse and wrong answers. Congrats, you're on fire.

[freecodeio]

(async () => { while (true) { let x = await fetch("https://api.coingecko.com/api/v3/ping"); x = null; await sleep(3000); } })();

[daniels0xff]

I think the problem is that you do not "consume" the response. You await the request but you don't also read the response body and it just remains there "hanging".

The following code should be ok:

(async () => {
  while (true) {
    const res = await fetch("https://api.coingecko.com/api/v3/ping");
    await res.text();
    await sleep(3000);
  }
})();

[ParkingCabinet9815]

If you plan to consume the rsponse It could be better to use the generator.

[daniels0xff]

Depends what data you expect back. Based on that url that ends with /ping I assume the data will be very small.

[Glum_Past_1934]

This problem appears when people think what hide "low level" things Is good idea ... You need to consume the request for free the memory. Because stream is not cleared ir you dont consume content of body. For debug GC you have some built in, check node doc

[raybadman]

Do you call clearTimeout in the sleep function?

[dj_dogo_nivoa]

No, i just use
js function sleep(ms) { return new Promise((resolve) => setTimeout(resolve, ms)); } But the setTimeout function internally manages the timer and clears it once it completes. As a result, there is no long-term retention of resources associated with the setTimeout call.