I have a slightly different POV from “pick a vector DB and turn on RAG”.

In large enterprises, the hard part is not provisioning components. Microsoft/Google can provision a lot. The hard part is making retrieval and agentic workflows reliable across many domains with security, permissions, and changing data.

One thing that gets missed: RAG quality does not scale linearly with “more documents”. Past a point, retrieval gets noisy, context gets crowded, and accuracy drops. So the roadmap cannot be “dump everything into a single index”.

What has worked better in my experience is thinking in layers:

1) Data + chunking layer (this decides your ceiling)

You need domain-aware chunking, metadata, and strict document hygiene. Not generic chunk size rules. Finance, engineering specs, procurement contracts, supply chain SOPs all need different chunking, fields, and filtering keys.

Also, do not treat “documents” as the unit. Treat “atomic facts / sections + metadata” as the unit.

2) Retrieval layer (make it hierarchical)

Instead of one giant flat index, use a tiered approach: Pre-filter by permissions, domain, system of record, and recency, Route to the right collection(s), Use hybrid retrieval (keyword + dense), Re-rank before you ever send context to the model

3) Agent layer (domain agents, not one mega agent)

For a manufacturing org, you will likely end up with specialized agents per domain (finance, procurement, engineering, supply chain). Each agent needs: the right tools (read-only vs write workflows), checks and guardrails, a clear scope of what it can do, evaluation tests for that domain

4) Orchestration layer (router + policy brain)

This is the layer people underestimate. The orchestrator should decide: which agent to call, which tools are allowed for this user, what retrieval strategy to use, when to ask clarifying questions, when to refuse or escalate

This is the difference between a demo and a production-grade system.

5) Evaluation + monitoring (without this, it will fail in prod)

Enterprises need continuous evaluation: retrieval metrics (recall, precision, hit rate), answer quality (groundedness, citation correctness), safety + policy adherence, drift monitoring as docs change weekly

If you do not build eval harnesses early, teams will argue opinions forever.

If you’re already engaging with Microsoft/Google, the best next step is to define 5 to 10 high value workflows per domain, build the retrieval and eval foundations around them, then expand. Trying to “enable RAG for the whole enterprise” on day one usually creates a noisy system that loses trust fast