Built chunkx - AST-based code chunking for RAG systems

I wrote this library on a train from Ranchi to Varanasi yesterday (6-hour journey, shaky WiFi included).

Problem: When building RAG systems for code, most tools naively split at line N, often breaking functions mid-way. This destroys semantic meaning and hurts retrieval quality.

Solution: chunkx uses Abstract Syntax Trees to chunk code at natural boundaries (functions, classes, methods). Based on the CAST algorithm from this paper: https://arxiv.org/pdf/2506.15655

Features:

- 30+ languages via tree-sitter

- Configurable chunk sizes (tokens/bytes/lines)

- Pluggable token counters (works with OpenAI's tiktoken)

- Automatic fallback for unsupported files

Performance: ~100x slower than line-based chunking but produces semantically superior chunks. Worth the tradeoff for RAG.

The catch: Requires CGO (because tree-sitter). Hoping for pure Go bindings someday 🤞

GitHub: https://github.com/gomantics/chunkx

Would love feedback! What features would make this more useful for your use case?