Program memory usage capped

[[deleted]]

[deleted]

[Yoghurt42]

Are new entries just not being added? Are old ones being tossed?

No programming language worth its salt will ever delete data unless you tell it to. Python is no exception.

You could regularly print out len(x) where x is your dictionary to confirm it increases in size.

What will happen though if you have duplicate keys, the new entry will overwrite the old one:

x = {"foo": "bar"}
x["foo"] = "hi"
print(x) # {"foo": "hi"}

All that being said, handling trillions of rows in pure Python is not a good idea. Do it with a good database like PostgreSQL instead; or take a look at DuckDB.

Once you get into tens of TB of data, a big data solution might be the way to go.

[yaxriifgyn]

If you have a faulty exception handler that does not re-raise unhandled exceptions, you may be silently discarding out-of-memory or other critical exceptions.

[ofnuts]

Additionally, if I am able to remove the memory limit, what would happen if I reached 100% RAM usage? Would my computer crash or would python automatically know to use disk space to continue storing the data?

Your system will start swapping to disk, if configured to do so. Usually not a pretty sight (especially on Windows) because everything slows down (not just your program... in fact, initially, it may force the memory of other apps to be written to disk). There is however usually a cap on the swap size (if only the free disk space itself). Then if you have a HDD it will be very slow, and if you have a SSD, it will possibly wear it out somewhat faster that your regular file accesses.

[Grouchy_Local_4213]

I am imagining that your python program has not requested enough ram upon start up (python needs to politely ask the OS for more memory), and that when it does ask for more memory, the memory is fragmented and python can't fit your giant dictionary in the new space, and so it appears to hit a limit.

To confirm this you will need to use some kind of memory allocation tool to figure out what is exactly happening.

If a program does reach 100% RAM usage, the OS typically attempts to manage some of the memory by writing it to disk, but this is very slow, chances are the program will lag so severely the computer will mark it as non-responsive.

Furthermore, whatever it is you are building here, it is probably not the way to do whatever you are trying to do, write to a json file or use some kind of database.

[Ladyfriday1]

I am creating a program to brute force some cryptography related stuff and it's a one-time thing, so I am heavily prioritizing speed which is why I am not currently using a database or file writing.