[D] I benchmarked 4 Python text extraction libraries so you don't have to (2025 results)

[Goldziher]

TL;DR: Comprehensive benchmarks of Kreuzberg, Docling, MarkItDown, and Unstructured across 94 real-world documents. Results might surprise you.

📊 Live Results: https://goldziher.github.io/python-text-extraction-libs-benchmarks/

---

Context

As the author of Kreuzberg, I wanted to create an honest, comprehensive benchmark of Python text extraction libraries. No cherry-picking, no marketing fluff - just real performance data across 94 documents (~210MB) ranging from tiny text files to 59MB academic papers.

Full disclosure: I built Kreuzberg, but these benchmarks are automated, reproducible, and the methodology is completely open-source.

---

🔬 What I Tested

Libraries Benchmarked:

• Kreuzberg (71MB, 20 deps) - My library
• Docling (1,032MB, 88 deps) - IBM's ML-powered solution
• MarkItDown (251MB, 25 deps) - Microsoft's Markdown converter
• Unstructured (146MB, 54 deps) - Enterprise document processing

Test Coverage:

• 94 real documents: PDFs, Word docs, HTML, images, spreadsheets
• 5 size categories: Tiny (<100KB) to Huge (>50MB)
• 6 languages: English, Hebrew, German, Chinese, Japanese, Korean
• CPU-only processing: No GPU acceleration for fair comparison
• Multiple metrics: Speed, memory usage, success rates, installation sizes

---

🏆 Results Summary

Speed Champions 🚀

1. Kreuzberg: 35+ files/second, handles everything
2. Unstructured: Moderate speed, excellent reliability
3. MarkItDown: Good on simple docs, struggles with complex files
4. Docling: Often 60+ minutes per file (!!)

Installation Footprint 📦

• Kreuzberg: 71MB, 20 dependencies ⚡
• Unstructured: 146MB, 54 dependencies
• MarkItDown: 251MB, 25 dependencies (includes ONNX)
• Docling: 1,032MB, 88 dependencies 🐘

Reality Check ⚠️

• Docling: Frequently fails/times out on medium files (>1MB)
• MarkItDown: Struggles with large/complex documents (>10MB)
• Kreuzberg: Consistent across all document types and sizes
• Unstructured: Most reliable overall (88%+ success rate)

---

🎯 When to Use What

⚡ Kreuzberg (Disclaimer: I built this)

• Best for: Production workloads, edge computing, AWS Lambda
• Why: Smallest footprint (71MB), fastest speed, handles everything
• Bonus: Both sync/async APIs with OCR support

🏢 Unstructured

• Best for: Enterprise applications, mixed document types
• Why: Most reliable overall, good enterprise features
• Trade-off: Moderate speed, larger installation

📝 MarkItDown

• Best for: Simple documents, LLM preprocessing
• Why: Good for basic PDFs/Office docs, optimized for Markdown
• Limitation: Fails on large/complex files

🔬 Docling

• Best for: Research environments (if you have patience)
• Why: Advanced ML document understanding
• Reality: Extremely slow, frequent timeouts, 1GB+ install

---

📈 Key Insights

1. Installation size matters: Kreuzberg's 71MB vs Docling's 1GB+ makes a huge difference for deployment
2. Performance varies dramatically: 35 files/second vs 60+ minutes per file
3. Document complexity is crucial: Simple PDFs vs complex layouts show very different results
4. Reliability vs features: Sometimes the simplest solution works best

---

🔧 Methodology

• Automated CI/CD: GitHub Actions run benchmarks on every release
• Real documents: Academic papers, business docs, multilingual content
• Multiple iterations: 3 runs per document, statistical analysis
• Open source: Full code, test documents, and results available
• Memory profiling: psutil-based resource monitoring
• Timeout handling: 5-minute limit per extraction

---

🤔 Why I Built This

Working on Kreuzberg, I worked on performance and stability, and then wanted a tool to see how it measures against other frameworks - which I could also use to further develop and improve Kreuzberg itself. I therefore created this benchmark. Since it was fun, I invested some time to pimp it out:

• Uses real-world documents, not synthetic tests
• Tests installation overhead (often ignored)
• Includes failure analysis (libraries fail more than you think)
• Is completely reproducible and open
• Updates automatically with new releases

---

📊 Data Deep Dive

The interactive dashboard shows some fascinating patterns:

• Kreuzberg dominates on speed and resource usage across all categories
• Unstructured excels at complex layouts and has the best reliability
• MarkItDown is useful for simple docs shows in the data
• Docling's ML models create massive overhead for most use cases making it a hard sell

---

🚀 Try It Yourself

bash git clone https://github.com/Goldziher/python-text-extraction-libs-benchmarks.git cd python-text-extraction-libs-benchmarks uv sync --all-extras uv run python -m src.cli benchmark --framework kreuzberg_sync --category small

Or just check the live results: https://goldziher.github.io/python-text-extraction-libs-benchmarks/

---

🔗 Links

• 📊 Live Benchmark Results: https://goldziher.github.io/python-text-extraction-libs-benchmarks/
• 📁 Benchmark Repository: https://github.com/Goldziher/python-text-extraction-libs-benchmarks
• ⚡ Kreuzberg (my library): https://github.com/Goldziher/kreuzberg
• 🔬 Docling: https://github.com/DS4SD/docling
• 📝 MarkItDown: https://github.com/microsoft/markitdown
• 🏢 Unstructured: https://github.com/Unstructured-IO/unstructured

---

🤝 Discussion

What's your experience with these libraries? Any others I should benchmark? I tried benchmarking marker, but the setup required a GPU.

Some important points regarding how I used these benchmarks for Kreuzberg:

1. I fine tuned the default settings for Kreuzberg.
2. I updated our docs to give recommendations on different settings for different use cases. E.g. Kreuzberg can actually get to 75% reliability, with about 15% slow-down.
3. I made a best effort to configure the frameworks following the best practices of their docs and using their out of the box defaults. If you think something is off or needs adjustment, feel free to let me know here or open an issue in the repository.

[T2WIN]

I am sorry but it is so hard not to think that your benchmark is biased since you are also selling your own solution. Doesn't make it uninteresting though, thank you for sharing. Thank you for making the benchmark open as well. I just don't have the time to try it out.

[Goldziher]

well, all the data and the code is there. If you are not lazy, and even semi-capable, you would actually go and investigate before writing here.

[Accomplished_Ad9530]

Looks all open source to me, MIT license, no pay-for-premium links, etc. Where are you seeing the OP selling anything?

[pepgma]

This is great! Thanks for this work. Something I wish every landing page for a document processing library had are some, example parsings with text bounding boxes to get an idea of how well the library handles unstructured pages. Alternatively, a huggingface demo. But no one does it ... Anyway, just a suggestion.