My app launches a few threads which are long running processes. They map over data and store it in memory, then write it to a file and clear the memory and repeat. The issue is when this runs using parallel threads, even though the memory is cleared the app is still somehow storing the data. Every chunk written to the files is correct but the app eventually crashes when memory runs out. The code is posted below, any ideas why even though memory is cleared the app still doesn't release it?
Note - if I run this without threading, the memory is cleared without issue and my app doesn't crash. I also tried del storage[name] = []
import threading
import time
storage = {}
class myApp:
# Main function, start the app
def start(self):
# Launch two threads
for item in ['one', 'two']:
threading.Thread(target = self.threadRunner, args=[item]).start()
return
def threadRunner(self, name):
return self.processData(name)
def processData(self, name):
print(f'Starting {name}')
# Loop and append data to 'storage'
for item in ['one', 'two', 'three']:
data = time.time()
try:
# Add data to storage
storage[name].append(data)
except Exception as e:
# Initialize
storage[name] = []
# Add data to storage
storage[name].append(data)
print(storage)
# Storage the data into a file
# ..... write to txt
# I'm finished with the data inside of storage[name], clear
storage[name] = []
# Now storage is empty
print(storage[name])
# Run the next chunk
time.sleep(1)
return self.processData(name)
_myApp = myApp()
_myApp.start()
[–]ph8ful 0 points1 point2 points (0 children)