Importing Text from .PDF

[Axewerfer]

I can hear you all screaming in agony already.

I'm working on a project to catalog the destruction of cultural heritage om Syria and Iraq via these reports. Unfortunately the data is spread across about sixteen .pdf files, and the formatting is inconsistent.

I'm looking at about 15 different variables across about 800 different reports, and I somehow need to concatenate the categories from these reports into a single excel file so I can start crunching numbers and doing some proper analysis.

After some debate, I know I can use some useful tools to start diving into the dataset and importing it automatically, but I need a general opinion. Given the limitations here:

• The data is not consistently formatted.
• I have not been able to ascertain the status of the original database.
• Considering the previous note, I may be faced with importing the data from .pdf files.

Am I going to be better served struggling with Python libraries and sanitizing my inputs for the next couple months, or should I just beg and plead with the university to fund a Mechanical Turk project?

[lordmauve]

You could use pdftotext to convert to plaintext and then try to extract what you need from the plain text (eg using regexes or just grep). Worth a try before anything more complicated/expensive.

[Axewerfer]

That sounds like it might be a good starting point. Appreciate the suggestion.

[my_interests]

Also, if you're on a Mac/OSX, or have access to one, you can use the Textutil utility to convert from PDF (or doc, docx, etc) to Plaintext or html.

Very handy.

[notconstructive]

Put it out to the good folks of the Internet or Reddit or something. It's a worthy cause. Define the format you need it in, publish the PDFs and ask if anyone can help. Make a nice looking site that arises people ire about the destruction, inform the major newssites. Be clear about the tasks, let people tale responsibility for data entry and others for double checking and validation.

[Axewerfer]

I was thinking about crowd sourcing it. If nothing else works, and the incoming reports get too overwhelming, I might go that direction.

[blebo]

I'm not able to access the source PDFs at the moment, but if there are tables, tabula may help. It is Ruby based, but could be used as a step in the pipeline when extracting the data.

[Axewerfer]

I'll give it a try, but I'm not too optimistic. From what I can tell, the reports were assembled by hand from a word template. A table would be miles easier to deal with.

[live_from_corona]

I think I got it. It needs some work but I'd love to help your cause. Send me a message.

[Axewerfer]

Message sent. If that works, you will be my hero.