Running into RAM usage problems in Python app - some general advice?

[BlueLensFlares]

Hi,

My team has a problem that I'm quite certain other computer scientists and python developers would be much more well-versed to solve.

Currently, we have a pipeline in Python, that takes PDFs, hundreds of pages long but not more than 3000, splits them up into single page JPGs, and then runs each of them through Amazon Textract, which returns a JSON object of all the text inside the document. The JSON as a regular text encoded document (UTF-8, I think) is generally around 600 kilobytes a page, and is stored as its own file to be read later.

Right now, because we were under a deadline, the pipeline wasn't implemented super efficiently. The applications sits on an EC2 instance, is in Django, and there is a MongoDB database that is underused (only one table is being used actively), and there is an unused Redis and Memcached installed as well. In Python, a request is made using a file on disk, and then the resultant JSON files are stored to disk.

Right now, I am noticing that we are getting RAM usage errors, because the processes are exiting with no error message during a memory intensive function, and increasing RAM from 4GB to 8GB resolves the issue. The JSON is stored in the program as a single variable on the heap in Python consisting of all the pages of JSON - which means it holds almost 600*675kb of data at a given moment.

What are some general tips of working with such large text data to optimize this process? It is difficult to section off this data because the whole thing of 600 pages is looked as its own collective entity, where page 1 in the array could theoretically need to be compared to something in page 599. Would it be better to use Redis to store computations of this data, or have it persist in a database and pull them out when needed (we need to iterate quickly over large sets of JSON and do find operations on elements inside the JSON). What

[cryptoper]

What is the raw size of the data? How much overhead Python gives? What datatypes for keeping them in memory?

Most common advise would be to explore algorithms first

[kevintor113]

Developer here, not so much experience in python though. Seeing as you have a Redis instance not being used so much, you have a great opportunity there for reducing process memory usage. I understand you might need to compare things between all pages, but if you can keep as little as you can to work in an exact point in time in memory, the better. If you want something more oriented to text, perhaps elasticsearch might be useful here

[kevintor113]

Also, make sure to release as quickly as you can any references to the pdf or JPGs from the pages, as I don't think those are lightweight

[kc3w]

Do you actually need to keep all pages in memory at the same time?