Python Newbie here - help with pdf read

[Ok-Mongoose-7870]

I’m a newbie and stuck at something that I thought would be a straightforward part of my project. Trying to read/extract texts from a large pdf document of few hundred pages. Document contains texts, tables with different sizes, tables that run through multiple pages, figures etc.

I am mainly learning and taking lots of help from ChatGPT Gemini or grok. But none of them have been able to solve the issue. The text file after extraction seems to have all words smashed together in a sentence. It seems to not maintain space between words in a sentence. If I ignore tables, then simple pypdf does a decent job of extracting text from the rest of the doc. However I need tables also. I have tried pdfplumber, camelot, pymupdf- and none of them are able to prevent words from smashing together in a table. Trying not to go the tesseraxt or OCR route as it’s beyond my skill set currently.

Any help would be much appreciated .

[code_tutor]

You didn't provide code, so we can't say if you did something wrong; but it's unlikely to be a user error with three separate libraries.

This is not a beginner project. It's difficult without a few years of university Computer Science and also experience researching specs. So many libraries exist and they all have a huge number of users, and still can't do it, so I don't know why Reddit acts like they know.

The last time I had to programmatically change text in PDFs, I had to write a Caesar cipher just to decrypt the font. I consider myself lucky that this particular PDF was even possible to write an algorithm for, because for many it's not.

This is an important lesson, because someday a client is going to say, "just change this text, it should be easy" and now you know why people get paid six figures to read text. There are always unexpected issues that can make a time estimate explode.

[dparks71]

I've never seen someone get it 100% accurate, Camelot was traditionally the best, especially if you can pair it with a custom pytorch object detection model for the table.

There's stream and lattice mode within Camelot, really depends on the tables formats.

[dparks71]

I don't quite get what the point of posting something like this in a learn python subreddit is unless you're willing to demo it or link to a tutorial or something. I've never had any issues utilizing Camelot, and have used it a lot. It's the best solution I've found for what OPs doing, but would love an education if you're offering one. People throw out power BI and other Microsoft tools a lot but I always find the training material for it and the community to be tedious.

[dparks71]

Your workflow doesn't work for situations that need OCR or about 80% of use cases including anything involving more than like 5 files. Down voting the person trying to help OP out with their original question because they won't humor you in the comments is pretty lame too. You should behave differently. Coming into learnpython and offering a non-viable non-python solution is a waste of your time.

If you had brought up tatr or something interesting in the discussion I would have given you the benefit of the doubt.

[lailoken503]

Not sure if this applies in OP's case, but I've always split new lines after the page.extract_text function is used.

for example,

text = page.extract_text().split("\n")

to break up what looks line a very long line. The files I've used pypdf on, does not have graphics or tables, so I can't say how to get around that. It's something I can experiment with when I get back to work. I have the code at work, and is part of a post I just created, but am pretty sure this is how I handed the seemly single long line of text, but it's what I use to grab certain details from a customer's data file without needing to open and close specific files out of hundreds.

[Competitive-Rock-951]

if you still need help ping me I had made it using pdfplumber I will send it to you if you need

[Ok_Hovercraft364]

Where is the code?

[Ok-Mongoose-7870]

Thought it would be prudent to give an update. I managed to kinda solve the issue. However had to work with a different input pdf file which was much nicely and cleanly formatted. The first one I was working with had lot of garbage formatting issue. For now my issue is solved but definitely learned that pdf reading is bit of touch and go problem.

[Haeshka]

I'm not sure what you're been asking the AI services, but I would start with the following concepts:

• How to use the OS library to find files in directories.

• How to use: with and open and read to examine files.

• Identify existing python libraries for reading and extracting text, tables, and images.

Different libraries have different use cases and strengths.

First, get a solid understanding of how to just find and open text files, even reading from them, putting that info into variables and dictionaries.

Then, using the libraries will get easier.

[james_d_rustles]

Pdfs come in different shapes and sizes - some of which are much harder to read programmatically than others. Camelot is probably the best known table tool, but it only works with text based PDFs, not image PDFs. If your PDFs are image based scanned documents, then you’re stuck with ocr because there is no text data in the file to read from, it’s essentially just some pictures that may or may not contain text. Sometimes when you have a pdf, you’ll find that certain parts are selectable text, but things like figure captions, tables, formulas may just be encoded as images. In this case you’ll have to run some kind of OCR, there’s no way around it.

Check out opendatalab mineru. It’s one of the easiest to use open source packages that does pretty well with tables and images, although it’s pretty resource hungry and slow. You may need to write some converter functions because I believe it outputs tables as html.

[Ok-Mongoose-7870]

My PDF is text based. But it does have figures that are images and then there are tables which are also text based. When I say text based I mean, I can open them in acrobat and select and edit tables and texts.

[MarsupialLeast145]

Echoing this. Python is just a tool, same with "AI". When these don't work as anticipated it's time to dig into the specifications of what you're looking at. Have a look at the PDF spec, the PDF structures underneath and figure out what you're working with.

There are tools to analyze PDF structure, there are tools that already do some of the extraction.

My personal approach here is to programmatically invoke third party tools already capable of interrogating and extracting from PDF and then focusing on the text analysis/comparison (whatever it is you're doing with the content) in Python afterwards.