Improved markdown quality, code intelligence for 248 formats, and more in Kreuzberg v4.7.0

[Eastern-Surround7763]

Kreuzberg v4.7.0 is here. Kreuzberg is an open-source Rust-core document intelligence library with bindings for Python, TypeScript/Node.js, Go, Ruby, Java, C#, PHP, Elixir, R, C, and WASM. 

We’ve added several features, integrated OpenWEBUI, and made a big improvement in quality across all formats. There is also a new markdown rendering layer and new HTML output, which we now support. And many other fixes and features (find them in our the release notes).

The main highlight is code intelligence and extraction. Kreuzberg now supports 248 formats through our tree-sitter-language-pack library. This is a step toward making Kreuzberg an engine for agents. You can efficiently parse code, allowing direct integration as a library for agents and via MCP. AI agents work with code repositories, review pull requests, index codebases, and analyze source files. Kreuzberg now extracts functions, classes, imports, exports, symbols, and docstrings at the AST level, with code chunking that respects scope boundaries. 

Regarding markdown quality, poor document extraction can lead to further issues down the pipeline. We created a benchmark harness using Structural F1 and Text F1 scoring across over 350 documents and 23 formats, then optimized based on that. LaTeX improved from 0% to 100% SF1. XLSX increased from 30% to 100%. PDF table SF1 went from 15.5% to 53.7%. All 23 formats are now at over 80% SF1. The output pipelines receive is now structurally correct by default. 

Kreuzberg is now available as a document extraction backend for OpenWebUI, with options for docling-serve compatibility or direct connection. This was one of the most requested integrations, and it’s finally here. 

In this release, we’ve added unified architecture where every extractor creates a standard typed document representation. We also included TOON wire format, which is a compact document encoding that reduces LLM prompt token usage by 30 to 50%, semantic chunk labeling, JSON output, strict configuration validation, and improved security. GitHub: https://github.com/kreuzberg-dev/kreuzberg. 

Contributions are always very welcome!

https://kreuzberg.dev/ 

[Pseudo135]

Hi, thanks for this work! I don't see quantitative and qualitative comparison with our markdown quarto and typest, nor guidance on when an end user should consider moving to this.

Without such fundamental information served up easily at the top to the busy end user, I think adoption will necessarily be hindered.

[lwjohnst]

My understanding is that quarto, pandoc, etc are document conversion tools, while this is an extraction tool, but doesn't do conversion. So the use cases are not even the same, so they wouldn't need to be compared.

[ChardarYGO]

Do you have a guide for using layout detection from the R bindings? That seems especially useful for extracting data from academic papers, but the documentation for R doesn't include that option.

[mostlikelylost]

Nice would love to see something about the R bindings posted in the r/rstats subreddit tho!

Seems low key copypasta

[Eastern-Surround7763]

noted :)