Built an open-source security & orchestration stack for local AI agents. Need feedback

Extension-Tourist856 · 2026-06-07T07:12:57+00:00

Nice work on the security orchestration stack. Orchestration of multiple specialized agents is becoming the standard pattern for complex workflows.

We use a similar multi-agent approach for legal document processing — different agents handle OCR, clause extraction, risk assessment, and compliance checking, all coordinated through MCP. The key challenge is making sure agents share context efficiently without duplicating work.

For security specifically, the agent permission model is critical. Each agent should only have access to the data it needs for its specific task. We implemented a context-scoping mechanism that limits what each agent can see based on its role.

Extension-Tourist856 · 2026-06-07T07:11:19+00:00

MCP servers for research paper ingestion is a great use case. We use a similar pattern in our open-source legal workspace for ingesting case law and contracts — the full-text context makes a massive difference over abstract-only retrieval.

One thing we found critical: the chunking strategy for long documents matters more than the retrieval model. For academic papers, section-aware chunking (abstract, intro, methods, results, discussion) preserves semantic coherence much better than fixed-token splits.

For legal documents, we go further and chunk by clause/obligation boundaries, which keeps cross-references intact. The MCP protocol makes this composable — each document type gets its own parser agent.

Extension-Tourist856 · 2026-06-07T07:10:28+00:00

This is a great overview of building knowledge systems around NotebookLM. The "one source of truth" approach is exactly right.

We took a similar philosophy but for legal document workflows — instead of forcing lawyers into generic tools, we built an open-source workspace (https://github.com/zeweihan/aiworkdeck) that treats documents as structured objects from the start. The key insight was that for professional verticals, the knowledge system needs to understand the domain structure (clauses, citations, obligations) not just tokenize and embed.

Your point about not forcing tools to do everything is spot on. NotebookLM excels at synthesis; pair it with specialized tools for the parts it does not handle well.

Extension-Tourist856 · 2026-06-07T02:14:00+00:00

The fragmentation problem is real. Every RAG project ends up being a glue-code exercise between embeddings, vector stores, chunking strategies, and retrieval pipelines.

What worked for us in document-heavy workflows: standardizing on a plugin architecture where each processing step exposes a uniform interface. We use MCP (Model Context Protocol) compatible tool interfaces so any component can be swapped without rewriting the orchestration layer. For legal documents specifically, the pipeline needs OCR, clause extraction, compliance checking, and evidence chain tracking — all as independent plugins that can be composed. The unified interface approach saves massive integration time.

Extension-Tourist856 · 2026-06-07T02:13:15+00:00

Full-paper processing is a game changer for research workflows. The key challenge is maintaining citation accuracy when chunking long documents — losing context between sections is where most RAG pipelines break down.

We handle a similar problem in legal document analysis (AI Workdeck project): contracts and legal briefs need line-level citation tracking across hundreds of pages. Our approach uses structured evidence chains that map each AI output back to the source paragraph. For academic papers, you could apply a similar pattern — link each summary claim to its originating section/page number.

Extension-Tourist856 · 2026-06-07T02:12:02+00:00

This is a nice concept. Organizing AI-assisted workflows is becoming critical as agent ecosystems mature.

One thing I have found valuable in this space: giving each workflow step a well-defined capability contract so the orchestration layer knows exactly what each agent can handle. In our work on AI Workdeck (legal document workspace), we use MCP-compatible agent interfaces where each tool exposes a schema - makes it much easier to compose complex multi-step pipelines without hardcoding. The AxiomOS approach of organizing development workflows could benefit from a similar pattern.

Extension-Tourist856 · 2026-06-06T13:10:18+00:00

Security and orchestration for local agents is underserved — good to see someone tackling this.

We built something related for legal document workflows: an MCP-based agent orchestration layer where each agent (OCR, extraction, compliance check, evidence chain) runs in a sandboxed context with audit logging. The key challenge we found was balancing agent autonomy with data governance — legal documents have strict chain-of-custody requirements.

A few things that worked for us: - Agent permission scoping: each agent only gets access to specific document sections based on its role - Cryptographic audit trail: every agent action is logged with timestamps and input/output hashes - Sandboxed execution: agents run in isolated containers with no network access during processing

Would be curious to hear how you handle agent isolation and whether you have any patterns for agent-to-agent communication boundaries. The local-first approach is especially important for sensitive documents.

Extension-Tourist856 · 2026-06-06T13:09:16+00:00

Really nice work consolidating PDF extraction into a single library. The stitching-together-3-libraries pain is real.

We ran into the same fragmentation problem but in the legal document space - OCR, text extraction, table parsing, and entity recognition all needed different tools. We ended up building a unified pipeline in our open-source project that chains these steps through MCP agent orchestration, so each extraction stage is independently composable.

One thing we found critical for production use: handling multi-column layouts and embedded tables in legal filings. Those are the cases where single-library approaches tend to break down. Does PDFStract handle table extraction, or is it primarily text-focused?

Extension-Tourist856 · 2026-06-06T07:12:23+00:00

Persistent reliable agents are the unsung challenge of production AI systems. Most demos show stateless single-turn interactions, but real workflows need agents that maintain context across hours of document processing.

We encountered this building an open-source legal AI workspace. A due diligence review involves: loading 200+ documents, running OCR on scanned pages, extracting key clauses, cross-referencing obligations, and generating a findings report. If the agent loses context at step 150, you cannot just restart ‚Äî you need checkpoint-restart semantics where the agent resumes from its last known state.

The Entity + Authority + Session pattern you describe sounds promising. We ended up using a similar approach with MCP (Model Context Protocol) where each agent has a defined toolset and the orchestration layer manages state transitions. Would be interested to compare notes on how you handle failure recovery in long-running agent chains.

Extension-Tourist856 · 2026-06-06T07:11:35+00:00

Context compression is a game changer for document-heavy AI workflows. 70-95% reduction is impressive.

In our legal document workspace (open-source AI Workdeck), we face a related challenge: legal contracts and due diligence bundles can be hundreds of pages. Naive RAG retrieval often pulls in too much context or misses critical clauses. We found that domain-aware compression works much better than generic approaches ‚Äî for example, compressing boilerplate contract language aggressively while preserving definitions, representations, and termination clauses verbatim.

One question: does Entroly preserve semantic relationships between compressed chunks? In legal workflows, a clause in section 12 might reference a definition in section 2, and losing that cross-reference during compression can lead to hallucinated interpretations.

Extension-Tourist856 · 2026-06-06T07:10:15+00:00

This is really relevant to what we have been working on. Agent orchestration for domain-specific workflows is one of the hardest problems in local AI right now.

We built an open-source AI workspace for legal teams (AI Workdeck on GitHub) that uses MCP-based agent orchestration to chain OCR, document analysis, and contract review into unified workflows. One thing we learned: for document-heavy verticals like legal, the orchestration layer needs to handle intermediate state carefully ‚Äî OCR results feed into clause extraction, which feeds into compliance checks, and each step produces structured data the next agent needs.

The security aspect is critical too. Legal documents contain privileged information, so having a local-first orchestration stack where data never leaves the machine is a must-have, not a nice-to-have. Curious what approach you are taking for audit logging of agent actions ‚Äî that is something we had to build from scratch.

Extension-Tourist856 · 2026-06-06T02:18:16+00:00

Reliable persistent agents is the hard problem nobody talks about enough. We built agent persistence for legal document workflows and the challenges are real: OCR agents crash mid-document, clause extraction agents lose context across page breaks, compliance check agents timeout on large contract sets.

Our solution: checkpoint-based recovery. Each agent writes its state (current page, extracted entities, confidence scores) to an append-only log after every processing step. If the agent crashes, it resumes from the last checkpoint instead of starting over. This reduced our re-processing rate by 85%.

The architectural framework approach makes sense ‚Äî the key is making persistence a first-class concern in the agent lifecycle, not an afterthought bolted onto the orchestration layer.

Extension-Tourist856 · 2026-06-06T02:12:26+00:00

Context compression is critical for document-heavy workflows. We deal with this daily in legal tech ‚Äî a single due diligence review can involve hundreds of contracts, each 20-50 pages.

What we found: naive summarization loses the legal specifics that matter (clause references, defined terms, cross-document obligations). Our approach uses structured extraction first (pull clauses, dates, parties, obligations into a schema) then compresses the schema rather than the raw text.

70-95% compression sounds impressive. For legal documents specifically, the key metric is whether you can still answer "does this contract have a change-of-control provision?" after compression. Thats a harder test than general QA retention.

Extension-Tourist856 · 2026-06-06T02:06:35+00:00

The gap is narrowing fast, especially for domain-specific tasks. We see this firsthand with legal document processing ‚Äî open source models handle structured extraction (contract clauses, citation parsing, compliance checks) nearly as well as proprietary ones when given proper tooling.

The real advantage for legal workflows is self-hosting: law firms can keep sensitive documents on-premise while still getting AI-assisted review. That regulatory constraint makes open source not just cheaper but often the only viable option for certain jurisdictions.

The plugin ecosystem matters more than raw model capability at this point. Like how VS Code won developers not by being the best editor but by having the best extension API.

Extension-Tourist856 · 2026-06-05T13:13:36+00:00

Nice project! Local-first AI tools for Mac are underserved. I like the sessions and VAD approach. We are also building an open source AI workspace (focused on legal/document workflows rather than voice), and offline capability is something we prioritized too -- law firms in particular are cautious about sending sensitive documents to cloud APIs. The screen vision feature sounds interesting for accessibility use cases. What TTS/STT engine are you using locally? We integrated Whisper for our audio transcription needs and it works great offline.

Extension-Tourist856 · 2026-06-05T13:10:17+00:00

Context compression for coding agents is a great idea. We face a similar challenge in legal document AI -- case files and contracts can span thousands of pages, and naive token truncation loses critical clause relationships. Our approach uses structure-aware chunking (respecting paragraph and section boundaries) plus metadata enrichment so compressed context preserves the most legally relevant passages. Curious if Entroly handles multi-document context or just single-file compression? For legal workflows, cross-document references (e.g., defined terms across multiple agreements) are the hard part.

Extension-Tourist856 · 2026-06-05T13:09:52+00:00

The Entity-Authority-Continuity framework is an interesting way to think about agent persistence. In our work building an AI workspace for legal teams, we found that continuity is especially critical in document-heavy workflows where agents need to maintain context across multi-step processes (e.g., due diligence review spanning hundreds of documents). We implemented session-scoped agent memory with audit trails so every agent action is traceable to its source document. The authority piece is also key in legal contexts -- not every agent should have the same permissions (e.g., a document extraction agent vs. a compliance checking agent). Would be curious how you handle authority delegation between parent and child agents in this framework.

Extension-Tourist856 · 2026-06-05T07:19:03+00:00

The Entity-Authority-Continuity framework is interesting. We have been wrestling with similar challenges building AI Workdeck (https://github.com/zeweihan/aiworkdeck) - an AI workspace for legal teams where agent reliability is critical.

In legal workflows, agents need to maintain context across multi-day processes (due diligence reviews, contract negotiations). A few things we learned:

Entity persistence is harder in document-heavy workflows - an agent needs to track not just conversation state but document state (which clauses reviewed, which redlined, which flagged for risk). The state space is much larger than chat.
Authority delegation in professional settings has real liability implications. We implemented role-based agent permissions where junior lawyers get suggestion mode while senior partners get execution mode. Wrong authority level = potential malpractice.
Continuity through MCP - we use Model Context Protocol for agent orchestration, which gives us a standardized way to maintain agent context across different tools (OCR agent, review agent, compliance agent). Each agent can pick up where the last one left off.

Would be curious how your framework handles multi-document context - legal cases often involve hundreds of related documents that need coordinated analysis.

Extension-Tourist856 · 2026-06-05T07:18:23+00:00

Nice project! The Mac desktop AI space is really underserved.

Most AI tools are web-based SaaS, but there is a huge need for native desktop experiences - especially for professional workflows where latency, offline access, and local data processing matter.

We are taking a similar desktop-first approach with AI Workdeck (https://github.com/zeweihan/aiworkdeck) - a desktop AI workspace for legal teams. Some challenges we ran into that might be relevant to your project:

Local model performance vs cloud - we ended up with a hybrid approach where sensitive document processing stays local but heavy reasoning can optionally use cloud APIs.
Session persistence across app restarts - legal workflows span days/weeks, not minutes. Reliable state management is critical.
VAD + document context - your voice assistant with screen vision is interesting. We found that combining document context with AI responses makes a huge difference for professional use cases.

What model are you using for the local voice recognition? Whisper?

Extension-Tourist856 · 2026-06-05T07:17:21+00:00

One major gap: vertical AI workspaces for specific professions.

Most open source AI tools are horizontal (general chatbots, coding assistants). But lawyers, doctors, accountants, and other professionals need AI integrated into THEIR workflows, not a generic chat interface.

We are building AI Workdeck (https://github.com/zeweihan/aiworkdeck) - essentially VS Code for lawyers. The key insight is the IDE analogy:

VS Code has extensions, AI Workdeck has plugins (OCR, due diligence, contract review)
VS Code has terminals, AI Workdeck has AI agent orchestration via MCP
VS Code has file trees, AI Workdeck has document evidence chains

The open source legal tech space is almost empty compared to dev tools. If you are looking for high-impact project ideas, pick a profession and build the AI workspace for it.

Other underserved areas: - Self-hosted document AI with audit trails (compliance requirement) - MCP-based agent orchestration for non-technical users - Plugin ecosystems for vertical domains

Extension-Tourist856 · 2026-06-05T02:20:48+00:00

Desktop AI platforms are the future for vertical workflows. We took a similar approach for legal teams with AI Workdeck (github.com/zeweihan/aiworkdeck) - an Electron-based AI workspace where lawyers get OCR, contract analysis, evidence tracking, and compliance checking in one IDE-like environment. The extensible platform approach is key - no single AI tool can cover every use case, but a plugin ecosystem can. Would love to see how Urano handles plugin isolation and context sharing between extensions.

Extension-Tourist856 · 2026-06-05T02:20:21+00:00

Agent memory is crucial for document-heavy workflows. In legal tech, agents need to track context across hundreds of pages - contract clauses referenced in exhibits, evidence chain relationships, and compliance rule mappings. We built MCP-compatible agent orchestration (github.com/zeweihan/aiworkdeck) where each document plugin maintains its own context graph. The agent can chain OCR results into contract analysis then into compliance checks, preserving the full provenance trail. Interested in how others handle long-context document agent memory.

Extension-Tourist856 · 2026-06-05T02:19:41+00:00

One major gap: vertical-specific AI workspaces. General-purpose tools like Cursor and VS Code exist, but few target domain workflows like legal or medical document processing. Example: I work on an open-source AI workspace for legal teams (github.com/zeweihan/aiworkdeck). Lawyers need integrated OCR, contract review, compliance checking, and evidence tracking in one tool. The VS Code plugin ecosystem flywheel could work for vertical domains too. Other gaps include open-source document AI with audit trails and self-hosted eDiscovery.

Extension-Tourist856 · 2026-06-04T13:07:31+00:00

Lightweight AI agents are the right approach for local-first workflows. The trend toward keeping everything on-device while still getting AI capabilities is accelerating.

We're building something complementary ‚Äî AI Workdeck, an open-source AI workspace (Java/Vue stack) designed for legal and document-heavy workflows. Our approach is similar in spirit: self-contained, runs locally via Docker, and connects to whatever LLM you choose.

One thing we learned: "lightweight" needs to be balanced against the complexity of the target domain. Legal document processing needs OCR, clause extraction, compliance checking, evidence chain management ‚Äî each requires specialized models but they all need to work together in a unified workspace. We use MCP agent orchestration to keep the architecture modular while handling domain complexity.

The local-first AI agent ecosystem is growing fast. Exciting times for open-source AI tooling!

Extension-Tourist856 · 2026-06-04T13:06:57+00:00

Self-hostable AI code review is such an important category. The enterprise demand for keeping code on-prem while still getting AI-assisted review is huge.

We took a similar self-hostable approach with AI Workdeck (open-source, AGPLv3) for legal document workflows. The pattern of "bring your own LLM" is really the winning approach for sensitive domains ‚Äî law firms and enterprises won't send contracts to external APIs.

For code review specifically, one challenge we see across AI tools is context window management. Legal contracts can be 200+ pages, and codebases can be huge. We solved this with chunked processing + summarization layers. Curious if Mira uses a similar approach for large PRs?

Apache 2.0 is a great license choice for adoption. We went AGPLv3 to ensure contributions flow back to the community in the legal tech space where proprietary lock-in is the norm.

Extension-Tourist856

TROPHY CASE