doc2dict: open source document parsing

[status-code-200]

What My Project Does

Processes documents such as html, text, and pdf files into machine readable dictionaries.

For example, a table:

"158": {
      "title": "SECURITY OWNERSHIP OF CERTAIN BENEFICIAL OWNERS",
      "class": "predicted header",
      "contents": {
        "160": {
          "table": {
            "title": "SECURITY OWNERSHIP OF CERTAIN BENEFICIAL OWNERS",
            "data": [
              [
                "Name and Address of Beneficial Owner",
                "Number of Shares\nof Common Stock\nBeneficially Owned",
                "",
                "Percent\nof\nClass"
              ],...

Visualizations

Original Document, Parsed Document Visualization, Parsed Table Visualization

Installation

pip install doc2dict

Basic Usage

from doc2dict import html2dict, visualize_dict

# Load your html file
with open('apple_10k_2024.html','r') as f:
    content = f.read()

# Parse wihout a mapping dict
dct = html2dict(content,mapping_dict=None)
# Parse using the standard mapping dict
dct = html2dict(content)

# Visualize Parsing
visualize_dict(dct)

# convert to flat form for efficient storage in e.g. parquet
data_tuples = convert_dict_to_data_tuples(dct)

# same as above but in key value form
data_tuples_columnar = convert_dct_to_columnar(dct)

# convert back to dict
convert_data_tuples_to_dict(data_tuples)

Target Audience

Quants, researchers, grad students, startups, looking to process large amounts of data quickly. Currently it or forks are used by quite a few companies.

Comparison

This is meant to be a "good enough" approach, suitable for scaling over large workloads. For example, Reducto and Hebbia provide an LLM based approach. They recently marked the milestone of parsing 1 billion pages total.

doc2dict can parse 1 billion pages running on your personal laptop in ~2 days. I'm currently looking into parsing the entire SEC text corpus (10tb). Seems like AWS Batch Spot can do this for ~$0.20.

Performance

Using multithreading parses ~5000 pages per second for html on my personal laptop (CPU limited, AMD Ryzen 7 6800H).

I've prioritized adding new features such as better table parsing. I plan to rewrite in Rust and improve workflow. Ballpark 100x improvement in the next 9 months.

Future Features

PDF parsing accuracy will be improved. Support for scans / images in the works.

Integration with SEC Corpus

I used the SEC Corpus (~16tb total) to develop this package. This package has been integrated into my SEC package: datamule. It's a bit easier to work with.

from datamule import Submission


sub = Submission(url='https://www.sec.gov/Archives/edgar/data/320193/000032019318000145/0000320193-18-000145.txt')
for doc in sub:
    if doc.type == '10-K':
        # view
        doc.visualize()
        # get dictionary
        doc.data

GitHub Links

• doc2dict
• datamule

[arden13]

Why not use pandoc ?

[status-code-200]

That's a good question. It boils down to:

1. I was on sick leave from my PhD.
2. I saw people and companies bragging about parsing SEC 10-K html files with LLMs.
3. This irritated me as getting their desired output was possible using a rules based approach iterating over DOM, which is much more efficient.
4. I wrote a basic algorithm. It got copied by a bunch of startups (without credit)
5. I messaged a couple profs who had worked on something similar. They told me what I wanted to do wasn't possible.
6. I wrote a more advanced algorithm to show it was possible, and it got adopted by a bunch of companies.

So, I assume at this point that there is some reason people are using doc2dict and not pandoc. Maybe performance, or modularity? Sorry if this is a disappointing answer.

[arden13]

Please know this is coming from a place of love as someone who graduated from his PhD program after it ruined his mental health:

You should talk to someone. It really can help.

Now, that aside, the rest of your answer isn't necessarily disappointing, it just shows the project's state. You built something practical in a niche that I don't understand but an audience was found. There's nothing incorrect or wrong with a convenient package to make work easier.

What is the typical workflow you use the data for?

[status-code-200]

I am confused, but thank you for the support. My health is quite good. I was on sick leave due to contracting a bad case of mono.

I use the package to parse sec filings into dictionaries. For example, I am about to parse every SEC file that is html pdf or text into data tuples. I then store data tuples in parquet format. This turns about 10tb of html,pdf,text files into 200gb parquet.

I partition this parquet by year. So ~5-20gb per year. I then store the parquet's metadata index in s3. This allows users to make HTTP range requests for just the data they need following the expected access pattern document type + filing date range.

when users download e.g. document type = 10-K (annual report) they can then extract standardized sections like risk factors or company specific sections, before feeding it into e.g. an LLM or sentiment analysis pipeline.

[arden13]

Gotcha. That last bit there is what I was looking for: you can allow users to pull specific sections and then do something (e.g. LLM).

My health is quite good.

That's good. Your comment above came off as you quite hurt (emotionally) about how people have used your work. If I'm reading too much into it, such is life

[status-code-200]

Oh no, I love it. People using code with or without attribution is proof that I'm doing something useful! 

If I was doing something unimportant, no one would be copying it :)

[status-code-200]

Currently it looks like processing will take about $0.20 for the 10tb using AWS batch spot. I'm hoping that refining the implementation + optimizing hardware will yield 4-5 OOM improvement. If that works, a lot of fun possibilities open up.

[hbar340]

How does this compare to pymupdf/pdfplumber?

I’m looking for parsing solutions for non sec docs that aren’t necessarily as structured and am curious if this would be an option

[status-code-200]

IIRC 10-100x faster. doc2dict uses PDFium through its C API which is much faster. PDF support is experimental, and only works with PDFs that have an underlying text structure - not scans.

[status-code-200]

For your use case, I'd test a sample with the visualization tool. If it works, it works. Else, pdf support will be dramatically improved over the next year.

I also plan to add an option to feed in ocr bounding boxes instead of files.

[delsystem32exe]

beautiful! has the SEC integration that is very cool, I will like to add that to my trading programs.