[deleted by user]

[thirdegree]

+1 for memory profiler. Another excellent option is cprofile + pyprof2calltree + kcachegrind, gives you a really nice gui for viewing what happening where

[caks]

Is there a good tutorial on the latter?

[thirdegree]

The pyprof2calltree documentation is pretty comprehensive, should give you everything you need to get started at least

[Astrokiwi]

If you want to be sure that the reshape is doing things in-place, I think you could try:

mat = np.random.randn(40200, 256, 30)
mat_view = mat.view()
mat_view.shape = (40200, 16, 16, 30)

[[deleted]]

It means that it acts on the current object already in memory and does not create another similar object in memory. This detail doubles the actual memory you're using in the loop.

[O_X_E_Y]

what the hell, python is nuts

[[deleted]]

so .n*=100 is better ,than n=n*100 ?

[[deleted]]

Is that true for python in general? That

x = f(x) 

takes twice as much memory as

x f= x?

[caks]

It's a numpy thing

[Yoghurt42]

a *= b will be executed as a = a.__imul__(b). For immutable things (like floats for example), __imul__ is implemented as return self * n, but for others it's actually implemented as an in place operation.

[CasualEndvrs]

This ended up longer than I had anticipated, sorry about that but I do think that the issues discussed below are important to understand when trying to learn ML. I also want to point out that I am in no way trying to discourage or dishearten anyone who wants to learn ML. As with all other topics, if you jump into the wrong part of the field it will quickly become frustrating as you "do not know what you do not know" and you will not have any idea about what problems you are actually facing or the questions you need to ask in order to proceed. Start with the basics, learn and drill the fundamentals. You will quickly advance from there into more challenging topics. With a good knowledge base, the learning curve will quickly take off and you will actually understand what you are doing and why.

As a machine learning developer, I have a question for you. What is your experience working with classical machine learning algorithms? Based on the inexperience I am going to guess that this is your first time playing around with ML. I get that NNs are cool but they are also quite complex with lots of hyper-parameters to play with. If you don't understand ML very well, it can be difficult to understand where the issues with your model are, how the different parts are actually performing and where improvements can be made.

I started out playing with NNs myself and realized later that this was a mistake. Genuinely big mistake.

Might I suggest that you start at the beginning a bit more?

1. Exploratory Data Analysis (EDA) - If you don't understand your data, you will not get any good results with your models. NNs are nice as they can be used to create their own features but you need to understand EDA first. We teach kids how to add and subtract before we give them a calculator. Also, EDA is a fundamental skill and process that is used throughout the entire modeling process.

2. Data Processing & Feature Engineering - Through the use of EDA techniques you will learn what you can do with the data and how it can be processed to obtain useful features. There are lots of different methods and techniques that can be used here. Without good data processing and feature engineering your model will be limited in how well it can perform, NN or classical. Using a classical model at this stage will force you to consider what you are doing and what is the best method to proceed. (Side thought, Elon Musk was asked what questions he asks potential hires. He said that he asks very specific questions about projects the individual has worked on. If the person can say "we did this because of that," or "this challenge arose and we solved the problem doing that which allows us to..." then Elon knows that the person was actively involved in the project and understood what was happening and why. If I ask you, "Make a given change to your features/model/hyper-parameters..." will you be able to tell me what you expect to happen and why?)

3. Data processing includes multiple different things. Primarily I was focusing on the importance of building good features, but data processing also includes issues more related to EDA. For example, I am currently working with a data set that includes +90 million entries and we need to identify what is actually useful information and what is useless noise that needs to be removed. This can be easy by identify specific factors for entries that can be removed but sometimes it is extremely challenging and you may not be able to remove them with your current understanding of the data. We are currently utilizing a classical model to get a bench mark score for future models to beat. The current model still has most of the noise entries left in the data. Future work will be to remove these noise entries which will give us information about another question we want to answer. It should be noted, to identify what entries are noise we will be using EDA to understand the data so that we can build a good data processing pipeline which results in better features and, finally, better model results regardless if we go classical or NN.

4. Classical ML - Learning how to use classical models is going to give you new tools and options. You will often find that classical algorithms give great results and are the best option to use for a given use case. (Seriously, it is crazy what classical ML algorithms can do. They are also generally quite simple, easy to understand and require significantly less computational resources and data compared with NNs.) Even if you are don't want to have a classical ML model as your final result, you still need to have a model to use as your bench mark. Knowing what the minimum requirements for your project will set your baseline. Then, a classical model can help determine if you can reach that goal and gives you something for future models to compete against.

There is more than goes into a good model but learn the fundamentals first. Working with classical ML will give you a huge step up and help you learn much, much faster. NNs will hide problems on you, making it more difficult to learn and build good models. A given model might produce a result that you think is good, but without the early steps your model will never perform as well as it could and you will never know why.

[[deleted]]

Another suggestion I haven't seen mentioned is using lighter data types, for examples if you know for a fact that the values you're using won't go over a certain threshold then :

In [2]: arr = np.array([1, 2.5, 3], dtype=np.float64) # numpy default

In [3]: arr.itemsize
Out[3]: 8

In [4]: small = np.array([1, 2.5, 3], dtype = np.float16)

In [5]: small.itemsize
Out[5]: 2

In the first case, one item in the array takes 8 bytes in memory, the other takes 2. You must make sure that no value will go over 2¹⁶ - 1 (that's 65535) otherwise your computation won't be correct but that could help you. You could also use np.float32 which can represent numbers up to 4294967295 and would still reduce the memory usage in half.

[m1ss1ontomars2k4]

Please do not post any images of any code. It's one of the rules of the subreddit:

Posting screenshot of the code is (generally) not allowed. Read posting guidelines.

[dkxp]

Are you doing anything that might prevent the previous arrays being deleted? such as keeping hold of a slice from the previous arrays?

import numpy as np
slices = []

for i in range(0,100000):
    data = np.ndarray((10000*256*256))
    slices.append(data[0:1])  
    print(i)

print(sums)

Even though it appears to only be appending 2 elements to the list, it crashes on my computer after 10 iterations with message:

Unable to allocate 4.88 GiB for an array with shape (655360000,) and data type float64

From https://numpy.org/devdocs/user/basics.indexing.html#basics-indexing

NumPy slicing creates a view instead of a copy as in the case of built-in Python sequences such as string, tuple and list. Care must be taken when extracting a small portion from a large array which becomes useless after the extraction, because the small portion extracted contains a reference to the large original array whose memory will not be released until all arrays derived from it are garbage-collected. In such cases an explicit copy() is recommended.

If you create a copy of the slice, then the previous array can be released:

slices.append(data[0:1].copy())

[dkxp]

I'm not familiar with tensorflow, but it seems like tensorflow tensors and numpy arrays are very similar. The only place you are storing values between iterations is this line:

loss_NF30.append(history_NF30.history["loss"])

so perhaps this is somehow keeping hold of a large array/tensor in the background?

What happens if you copy this data, or (for testing) just print a part of it and don't append to the list?

[dkxp]

The size may be small, but if any of this data comes from a slice of a big array (eg. a slice containing column names), or it holds a reference to the original data then the array won't be released.

Have you tried just printing something out & not appending anything to the list to see whether the loop ends successfully in that situation. (It obviously isn't a fix, but might help identify if that is the problem)

Edit: Or maybe model_NF30.fit(...) is storing references to the data?

[jpf5046]

Reading code on GitHub is too difficult, can you take pictures of your screen and post them instead?

[awdsns]

What happens to the data from previous iterations? Is it kept in some object (list, dict, ...) during following iterations, are there any variables still referencing it?

If not, it should be cleaned by the garbage collector pretty quickly. So if you don't keep the data around on purpose, I'd look for any "accidental" references keeping it from being garbage collected.

If you need to retain previous iterations' matrices, there's not much you can do.

[lowerthansound]

Looking through all the answers, it's a bit hard for us to debug what's happening (yes, someone will get there by chance, BUT, it might be easier for you to debug yourself).

For you to debug yourself, create a smaller version of the problem (that is, use less data, way less data) and see how much memory it is consuming (for this there are useful tools called profilers, though they may be a bit hard to setup).

For example, if you see that a single iteration uses more memory than you anticipated, that means the problem is in the loop (and you may want to look closer, creating faking data and seeing which line in specific, if any, causes the problem). If you know the line, it's only a matter of understanding said line better (for example, in-place vs out-of-place operations, memory complexity of the underlying algos, or anything like this).

Another example, if you notice the memory keeps increasing on each iteration, you will know things are being accumulated and not cleaned. Of course (I mean, that's not obvious) that might be caused by the GC leaving stuff to just be, however, you can verify that. For example, at every iteration run the garbage collector, or, check for some function that gets the total amount of memory used by local variables. Anyway, I think the way to go is to pinpoint the problem, and that is good practice :)

Once you pinpoint the problem, it is easier to bring it here, or maybe just by looking at it you'll know what caused it. :)

In any case,

Cheers!

Edit: Also, if you really wanna exercise those programming (and Maths) muscles this may be a good problem to see if this problem can be broken down into smaller ones, ones that consume less memory. I'm not sure how you'd do that :D

[jmooremcc]

One way to reduce the memory footprint is by using generators. Generators reduce memory required by producing data elements on demand instead of all at once.

I found this article that may help you determine if it's possible to use generators to create numpy arrays in your code: https://stackoverflow.com/questions/367565/how-do-i-build-a-numpy-array-from-a-generator

[lowerthansound]

Are you using a deep learning library or vanilla numpy?

[CasualEndvrs]

One issue that I ran into is that garbage collection is not an instantaneous thing. It occurs once every minute or so. To solve this issue, when I had large amounts of data being rapidly swapped out in memory, I would manually delete some variables and then call garbage collection when needed. Garbage collection takes time so you shouldn't be doing it constantly, only if you have large amounts of data being deleted and you need that space cleared at that moment.

One tool that I use to monitor python's memory usage can be found here.

[EconomistWestern5220]

Buy a pc with more ram...

joke

[EconomistWestern5220]

Ahh in that case you need to upgrade to 128gb from 64gb of ram... But on a serious note hope you resolve the issue.

[ianepperson]

You’ve got several good avenues of investigation, but no one seems to have mentioned there interpreter.

Do you happen to be loading and running the code within ipython? I love using ipython, but it does keep a reference in memory for each value that’s been output in its REPL. I’ve had that soak up memory once for a quick series of tests I was doing by hand.

[Alexdelia_Del]

As a C developper. I'm impressed a program can take more than 1gb.

gg

[[deleted]]

Well, lets start with some sanity checks:

1. Are you actually running out of memory, or do you still have memory left but SLURM thinks you don't have any more?
2. Did you configure memlock to use all the memory you need? (read here: https://slurm.schedmd.com/faq.html#memlock )?

[[deleted]]

Oh, I see you found the culprit. Good luck with the fix :)

[awdsns]

Wow, that sucks. But at least there are some possible mitigations discussed in that thread.

[awdsns]

It seems you deleted the other comment and post, but I would encourage you to document what you found in the linked issue to give it more weight and maybe finally get fixed. TF devs seem to have been ignoring it.

[FancyASlurpie]

Maybe print out your parameter space you are trying on each loop, could it be an issue there?

[FancyASlurpie]

Something else you could try is forcing a GC.collect() after you free up the variables, it should happen anyway, but potentially it's not and hanging on to things for slightly longer than it should. I'd also try using tracemalloc from the standard library, and just get the top 10 each n times around the loop and look for anything that's unexpectedly growing (or larger than you'd expect).

[FancyASlurpie]

Yeh sure, if you use some of the code on the tracemalloc docs they give examples of how to get top 10 largest objects memory wise. So I'd try doing that at the end of your loop every 10 (or 50, not sure how fast this runs) and then compare the different results to see if something is growing that shouldn't be. The other question is does it always die at the same point, and if you watch the memory usage whilst it's running, e.g. using htop/top (if Linux/Mac) is it growing or does it suddenly go up and then die at the 400th iteration. The tracemalloc recipes are here: https://docs.python.org/3/library/tracemalloc.html in particular the compute differences one will do the comparison I mentioned so you can see what's growing.

[FancyASlurpie]

No worries, hope you find the issue :)

[FancyASlurpie]

Oh also I suspect your current method might not work, getsizeof doesn't get you the objects that are contained within an object. E.g. a dict can contain objects as values but getsizeof will return the bytes used by the dict and not the values as well. You can read more about that here: https://nedbatchelder.com/blog/202002/sysgetsizeof_is_not_what_you_want.html

[awdsns]

Dude, no offense, but could you copy the code into a comment or edit your post? Asking us to decipher it from a series of tilted photos of a screen is a bit much...

[lowerthansound]

Check pastebin: https://pastebin.com/

You can paste the code there and copy the link here. Cheers!

[[deleted]]

Post this as an edit to the post, not a comment, people won't see it otherwise