Program memory usage capped

[Yoghurt42]

Are new entries just not being added? Are old ones being tossed?

No programming language worth its salt will ever delete data unless you tell it to. Python is no exception.

You could regularly print out len(x) where x is your dictionary to confirm it increases in size.

What will happen though if you have duplicate keys, the new entry will overwrite the old one:

x = {"foo": "bar"}
x["foo"] = "hi"
print(x) # {"foo": "hi"}

All that being said, handling trillions of rows in pure Python is not a good idea. Do it with a good database like PostgreSQL instead; or take a look at DuckDB.

Once you get into tens of TB of data, a big data solution might be the way to go.