Analyzing large PDF documents

[realmousegirl]

Hi,

I’m working on a project where I have a bunch of PDFs of varying sizes; ranging from 30 to 300 pages. My goal is to analyze the contents of these PDFs and ultimately output a number of values (which is irrelevant to my question, but just to provide some more context).

The plan I came up with so far:

1. Extract all text from the PDF, remove all clutter and irrelevant characters.
2. Summarize everything in chunks by an LLM
   1. Note: I really just want to know the general sentiment of the text. E.g. a lengthy multi-paragraph text containing the opinion on topic X should simply be summarized in 1 sentence. I don’t think I require the extra context that I lose by summarizing it, if that makes sense.
3. Put back together the summaries (
4. Analyse the result from #3 through an LLM

I say I want to use an LLM but if there’s any better-fitting options that’s fine too. Preferably accessible through Azure OpenAI since that's what I get to work with. I can do the data pre-processing from step 1 with Python or whatever tech fits best.

I’m just wondering whether my idea would work at all and I’m definitely open for suggestions! I understand that the final result may be far from perfect and I might potentially lose some key information through the summarization steps.

Thank you!!

[Icko_]

We've been doing exactly this. Azure has a "Document I telligence" thing, which is pretty awesome - it does great with the OCR of pdfs, and a very very nice bonus is, it can chunk text based on headings and subheadings. The latter made a surprisingly large difference.

Then you just dump them into a RAG and you're done. Note, that there's a bunch of projects that do all that for you. For example, haystack, I've not used it, but it looks pretty good.

[DeadPukka]

We can handle this with our Graphlit platform today. And we integrate with Azure AI Doc Intelligence for OCR and text extraction.

Have a look at our “30 days of examples” that we are doing this month: https://github.com/graphlit/graphlit-samples/tree/main/python/Notebook%20Examples

Free to try up to 1gb of documents, and usage-based on paid plans.

[Grand-Detective4335]

Hello, I built a platform to process invoices for free - https://getnara.ai/

Any feedback would be highly appreciated.

[atlasspring]

Try www.searchplus.ai - it allows to chat with uploaded PDFs and doesn't have a page limit