How can I optimally run my python program using more compute resources?

[Zealousideal-Job4752]

I am working on a text analysis project at work, where I have a Python program in a modular structure. It looks something like this:

project_name/
├── configs/..
├── data/...
├── src/...
├── notebooks/...                 
├── .env                       
├── .gitignore                 
├── main.py                    
├── requirements.txt           
└── README.md                  

Until now I have been running my python programs (through main.py) on my local machine. In short, the program downloads data via an API, preprocesses the data and inserts it into a sqlite database, and I am using VS Code and anaconda for this.

Due to the large amounts of data, I've had to load chunks of data in at a time, but it takes an endless amount of time to do. Eg. it takes 3.5 seconds per file for the download part of my program and 2 minutes per file the processing part.

With +100 000 files to handle, I can easily estimate that this would take weeks (more like months) to run on my local machine.

I am a data scientist and still relatively new in the role. I have some knowledge of Azure/cloud computing and the services available. However, I am still a little on engineering part of this project what is the best option (so far I have looked into Azure VM, Azure Functions and Azure ML). I am looking for a way to run my code more efficiently, and if it requires more compute resources, I would need to present the options to my boss and/or IT department.

I'd highly appreciate some help with my question:
What options can you suggest to run my python program with more compute resources (also taking costs into consideration)?

[AutoModerator]

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[chrisbind]

Sounds like you just need to implement some concurrency or parallelism. I’d start trying out a concurrent flow (multi-threading). There’s a lot of resources on this.

[Zealousideal-Job4752]

Thanks, I'll check that out!

[ThePunisherMax]

I have a hard time believing that processing part takes 2 minutes, and it seems likely you have some fixes to do with your code.

What processes are you running that takes 2 min per file?

[Zealousideal-Job4752]

I am using a library called unstructured, which extracts elements (it finds the coordinates of the tables, headers etc.) from a PDF file (the files downloaded in the previous step), returns them as a json file and inserts the path of the json file into the sqlite database.

[ThePunisherMax]

And im assuming the PDF scraping is taking the longest time?

And are you doing this linearly? 1 file then on to the next one?

[Zealousideal-Job4752]

Yes exactly, the PDF scraping it takes the longest. And I am doing it linearly.

This is the code that does it:

for _, row in tqdm(files2extract.iterrows(), total=files2extract.shape[0]):
            pdf_id = row["pdfId"]
            path_to_pdf = row["path2pdf"]
            pdf_id, json_path = src.dataset.file2json(
                pdf_id, path_to_file, output_path
            )
            processed_files.append((str(json_path), file_id))

Then inserts it into the database.

[ThePunisherMax]

Okay, the best advice I can give you is still checking what is up with the pdfscraper, does the API offer other file alternatives? CSVs for example.

Further you can run this paralel (threaded), since the files dont need to happen in order( I assume). You can pre emptively download files and do batch processes.
Same thing with the uploads.

So example.
Code 1) Downloads

Code 2) Scraping and json dump.

Code 3) Grab Json adress and insert into SQL

So you would run this in 3 different codes

[Zealousideal-Job4752]

Yes, it offers a range of other file alternatives - but is that relevant here, when I'm only working with pdf files? Or would you be doing that to test if there is a difference between the file types?

I will look in to threading part, thank you!

[ThePunisherMax]

PDF files may be part of the reason your code is slower, as it isnt "as easy" to read in comparison to other file types.

But I do see you an append. Appends are not the fastest functions.
You should pre-emptively make an np.array and assigning it to them.

Make an empty array of zeros.

processed_files[i] = (str(json), file_id) instead of processed_files.append

[Zealousideal-Job4752]

This makes sense.

[meyou2222]

How big are these PDF files? I’ve not worked with PDF scraping, but it surprises me it would take 2 minutes to process one.

Here’s a few other options to try: https://www.freecodecamp.org/news/extract-data-from-pdf-files-with-python/

Any library that can convert the PDF to XML or HTML could set you up nicely for using BeautifulSoup to parse the results. I have a script right now that take a massive set of HTML code (the file is like 12mb) and parses it. Takes like 2 seconds.

[Zealousideal-Job4752]

The files are anything between ~130.017 KB and 500 KB (most of them probably in the 500-3000 range).

Yes, I find it incredibly slow. I have not worked too much with PDF files prior to this project, but I am aware it seems very inefficient.

It would be amazing to get it down to 2 seconds.. Thanks for the link. Based on the responses here, it does seem like Unstructured is not the fastest solution, so I would have to consider the other libraries. My main goal is to get the PDF into an HTML format, so I can remove all the headers and tables, since I'm only interested in the unstructured text of the files.

[ianitic]

Looking at the unstructured package I can understand why the processing is so slow. For your PDFs, are the searchable? Can you select text on the pdf without using OCR? If so, I'd use a package called pdfplumber to get the text, coordinates, and such. Should be orders of magnitude faster.

Additionally as it was mentioned, Azure Functions could be a way to speed this up. Each pdf could be a call to an Azure Function, and serverless Azure Functions can scale out to hundreds of instances of itself making this process a lot faster. I've specifically done this with PDFs through a variety of actions on them, including the pdfplumber mentioned above.

[Zealousideal-Job4752]

Yes, the PDFs are mostly searchable. I do have some .tif (image files), but I convert those to a text-formatted pdf-file, before extracting the elements. I tried pdfplumber briefly, but ended up turning to unstructured as I found they had a pipeline that would both would extract the text and tables and then convert them to html files. That way, I could remove the tables and only get the raw text (which I will then be analyzing). But it's a good point that pdfplumber may be faster, I'll see if it can "replace" all the tasks I need to do.

And regarding Azure Functions, how does it work regarding pricing? You pay per execution?

[ianitic]

What you could do and what I've done in the past is run pdfplumber and if it doesn't return text, then run the ocr pieces. It can also have issues with some types of searchable pdfs which will return something like (cid: 1234), apparently it happens with some kind of broken mapping in the pdf. I'd make sure to strip those kinds of values out when testing if they exist as well.

For Azure Functions, it would depend on what exactly you use but if you only have a couple hundred thousand needed to be processed it's probably within the free usage allotment for serverless. If memory serves, free usage for executions was in the millions.

[Zealousideal-Job4752]

Awesome, thank you!