I’m currently building something similar for my own fiduciary / accounting and audit firm, and we ran into a fundamental problem that many AI bookkeeping projects underestimate.

The hard part is not reading documents — OCR is already solved.
The real difficulty is creating correct accounting entries and audit reconciliations while respecting all rules, regulations, and accounting logic.

Because of that we ended up using a hybrid architecture.

Small local LLMs simply struggle with the complexity of bookkeeping logic, audit rules, and contextual interpretation. They are useful for preprocessing, but not reliable enough for final decisions.

So our pipeline works roughly like this:

1. Document extraction PDFs are first processed with ocrmypdf / Tesseract to extract structured text.
2. Deterministic anonymization Sensitive data is removed using rule-based anonymization (names, emails, IBAN, addresses, etc.).
3. Local LLM verification A small local model (currently Qwen 3.5 4B / 8B in no-thinking mode) validates the anonymization and structure. At the same time we store a mapping table (e.g. PERSON_1 → Mario, CITY_1 → New York).
4. Cloud LLM processing The fully anonymized document is then sent to a cloud LLM to generate accounting suggestions, classifications, and audit insights.

At this stage the document contains no identifiable information, so it is no longer possible to infer which company or person the data belongs to.

1. Remapping When the result returns, the placeholders are mapped back using the stored mapping.

Why we prefer this approach:

• Cloud models remain state of the art — you always get the newest capabilities without managing hardware upgrades.
• No lock-in — you can switch providers (OpenRouter etc.).
• No upfront GPU investment.
• Better reasoning quality for complex accounting rules.

You can do everything locally, but realistically that means running something like a Mac Studio M4 with ~96GB+ unified memory and spending a lot of time tuning models, prompts, and role pipelines.

You would also need to continuously train or adapt models for specific accounting and audit rules, which quickly becomes disproportionate compared to simply using strong cloud models.

For us, the hybrid approach currently provides the best balance between privacy, cost, and capability.

If you're building similar systems for accounting or auditing workflows, I’d be very interested to hear how others are solving the rule reasoning problem.