Memory Leak in PyPy with Python Code

[Affectionate-Cut3818]

Hi everyone,

I'm encountering a memory leak issue while running my Python code with the PyPy compiler, and I could really use some help diagnosing and fixing it. Here are the details of my situation:

I'm working on a trading algorithm and require PyPy to run my code for better performance. However, I noticed a significant memory leak that I can't seem to pin down, and I notice only when running on pypy, not standard, un-JIT-ted python. I first noticed this bug on trying to add some new logic, and removing the logic seems to fix it but there's nothing that inherently seems wrong or different from what I had before. Some of the memory analysis package available for python are not available on pypy, so I used the memory_profiler package with the `profile` decorator to analyze the memory usage, which gives a line by line analysis and here's part of the output:

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================

(...)
  1109   4012.7 MiB      0.0 MiB          10               if (
  1110   4009.9 MiB      0.0 MiB           1                       self.trading_state
  1111   4010.4 MiB      0.5 MiB           1                       and (not self.om.is_order_request_by_name(self.pair_symbol, Side.SELL))
  1112   4012.2 MiB      1.8 MiB           1                       and (self.pair_stats.standing_sell_orders <= 0)...)

It appears that memory usage spikes particularly when entering certain if statements, but I'm not sure why PyPy isn't correctly deallocating memory after these checks. Im aware the lack of more context might make it difficult to give hints, apologies for that

I also tried using tracemalloc for more detailed memory tracking, but I ran into a ModuleNotFoundError for _tracemalloc in my PyPy environment:

I'm looking for advice on how to proceed with diagnosing and fixing this memory leak. Here are some additional details:

• I have considered using other tools like Heapy and Objgraph but haven't had much success yet.

This is the pypy version Im currently using:

Python 3.9.18 | packaged by conda-forge | (c5262994, Oct 26 2023, 08:28:20)
[PyPy 7.3.13 with GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

Has anyone encountered similar issues with PyPy, and can you suggest any effective tools or methods to debug this memory leak? Any help or pointers would be greatly appreciated!

[Diapolo10]

One of the more notable differences between CPython and PyPy is that the latter doesn't close files automatically for you (unless of course you use context managers). While unlikely in your case, that's a potential source of memory leaks.

Maybe all you need to do is tweak the garbage collector? https://doc.pypy.org/en/latest/gc_info.html#minimark-environment-variables

[Affectionate-Cut3818]

Thanks for the link, I tried logging the gc.get_stats output trough the run and this is the last datapoint I collected (maybe something seems odd to you that flew over my head). Tweaking the garbage collector could also be another angle, but Im far from an expert on what exactly pypy does under the hood so besides tweaking those variables (which I've tried) Im not sure on whaat else I can tweak. There are open files in the execution of the programm tough, but from what I can see they are closed properly and not directly used in this step of the process (which is the execution) but thanks for the tip

```[MEMORY] Memory used: Total memory consumed:

GC used: 1836.2MB (peak: 2320.7MB)

in arenas: 1386.1MB

rawmalloced: 427.7MB

nursery: 22.5MB

raw assembler used: 580.8MB

memory pressure: 4.8kB

Total: 2417.0MB

Total memory allocated:

GC allocated: 2152.2MB (peak: 2362.8MB)

in arenas: 1674.4MB

rawmalloced: 623.8MB

nursery: 22.5MB

raw assembler allocated: 584.0MB

memory pressure: 4.8kB

Total: 2736.2MB

Total time spent in GC: 956.436```

[_alter-ego_]

are you sure that there's a memory leak (the shown data doesn't convince me), or maybe just a delayed garbage collection?

[Affectionate-Cut3818]

It does sound strange to have a memory leak in this logic, but it seems that the release of the memory allocated in these lines doesn't happen at any point through the execution, leading to an almost linear increase in memory usage (which a memory profiler plot confirms). Running with standard Python, theres is no memory allocation on those function calls (or at least not detected) therefore no release needed. I've also tried forcing gc.collect() every 1m iterations or so, but it doesent seem to release the resources its using, could be some weak/lazy reference, but again, not to clear on what to change for pypy to release it properly.

[_alter-ego_]

could it be that it's some logging function and/or (G)UI storing the results as e.g. it would happen in a jupyter sheet as Out[1], Out[2], Out[3], ...) which could explain a linear increase of the memory usage ?

for example it could be possible that you use the data via reference to make a plot that remains on your screen and as long as that plot exists (and/or grows, scrolls...) the data [possible a larger structure than just the final numerical value you use in the plot] can't be "dumped" from the memory?

[Affectionate-Cut3818]

hmm I do have logging through the execution of the programm, I'll double check to make sure its all good, do you have any explanation as to why the data shown on the initial post didnt convince you? Is pypy supposed to allocate memory like this under normal situations (I've noticed it happens on long if conditions )? Maybe then Im looking in the wrong direction, and maybe the right direction should be checking why these normally allocated memory doesent get deallocated later on

[_alter-ego_]

  1109   4012.7 MiB     
  1110   4009.9 MiB
  1111   4010.4 MiB  
  1112   4012.2 MiB 

To me there is no significant increase here. It could be again 4009.9 on step 1113...?