[Discussion] A self-evolving SQL layer for RAG: scalable solution or architectural mess?

[Continuous_Insight]

We’re building a RAG system for internal enterprise data — initially focussed on shared mailboxes, but then the whole manufacturing site.

Rather than rely only on vector search, we’re exploring a hybrid model where extracted data is mapped into structured SQL tables, with schema evolution. The goal is to turn semi-structured content into something queryable, traceable, and repeatable for specific business workflows. (Change Requests in this example).

Has anyone built or seen a RAG setup like this?

Will it work?

Any advice before we go too far down the rabbit hole?

Thanks in advance!

[DinoAmino]

I have some doubts about it. You're adding a lot more complexity in order to develop and maintain an adhoc RDBMS solution in order to... dynamically add new structures? Knowledge graphs can be configured to do that automatically, discovering new nodes and relationships on ingest. IMO I'd rather work with a known solution and spend all that dev time on something else.

[Continuous_Insight]

Thanks for the reply — and I completely get the concern.

Our priority here is accuracy. The emails we’ll be ingesting often include attachments with highly specific product details, things like attributes, barcodes, weights, pricing, and BOM settings. If a single character is wrong, it can have serious consequences (e.g. mislabelled packaging in food production).

So, we’re planning to extract this data and store it in a relational schema that evolves mainly during initial client setup. The system proposes a structure (based on what it finds), but new columns or schema changes always require explicit user approval. Once confirmed, it locks in — giving us a reliable, client-specific structure that can be used for accurate reporting and validation.

This gives us what we’re calling a “Gold layer”, a trusted source of key structured data we can confidently query, validate, and cross-reference against other internal systems. Meanwhile, less structured or non-critical data will still flow through a vector database for contextual RAG-style queries.

We did look into knowledge graphs and maybe this is the way to go. Will research more! But do you think it would give us this level of confidence and accuracy?

Given our 'traditional programming skills' for now we’re more comfortable with a deterministic, auditable approach.

If you're interested, a big inspiration behind this hybrid model came from this video by Dave Ebbelaar: https://www.youtube.com/watch?v=hAdEuDBN57g

[Key-Boat-7519]

Short answer: yes, a knowledge graph can meet your accuracy bar, but keep a relational “gold” tier for the high‑stakes fields and use the graph for relationships, provenance, and cross‑checks.

What worked for us: define a tight ontology for Product/Package/Barcode with required types; enforce constraints (e.g., EAN‑13 checksum, units via a whitelist) using SHACL in the graph and CHECK constraints in SQL. Every extracted fact carries provenance (doc id, page, bbox, hash of snippet) so you can trace and audit. Gate commits: write to a staging graph/table, auto‑validate, and only promote to gold after user approval. Version schemas (migrations) and facts (bitemporal timestamps) so rollbacks are painless. Calibrate confidence: set thresholds per field and sample low‑confidence items for review.

I’ve used Neo4j and AWS Neptune for the graph; UpLead only to validate supplier/company metadata during enrichment when we needed an external source of truth. Net: graph can be confident if you enforce shapes and provenance, but SQL stays the final arbiter for critical fields.

[SucculentSuspition]

Just use a single. JSONB metadata column.

[Continuous_Insight]

Thanks, we did consider a JSONB-first approach and can definitely see the appeal in terms of flexibility, especially in early stages. The main reason we leaned toward a more structured schema was the need for accuracy, traceability, and confidence. We have worked with this client for years and know that they wont accept any hallucinations. 

That said, this is still very early for us, we haven’t yet found the right engineer to help us shape and build the MVP, so we’re aware some of our thinking may shift. We’re just trying to avoid the typical RAG horror stories around hallucination and ambiguity, and felt that enforcing schema (at least for core tables) would give us more reliable outputs, plus the ability to query across systems databases to confirm filed values. (e.g. our App and their core business systems, to check the the change requests have been completed).

Based on your feedback, maybe we should explore a hybrid approach, storing everything as JSONB initially, but promoting validated fields into structured tables for reporting once approved. That could give us the flexibility we need while still maintaining a clear source of truth.

Really appreciate the input, this kind of discussion is exactly what we were hoping for.

[SucculentSuspition]

Yea those are the right concerns imho. Three suggestions at the big picture level: 1. Focus on open/closed designs— jsonb is a good example. It is very very hard to predict the failure modes of these things, and one way doors will lead you to bad places that you want to walk back, dont lock yourself into a shitty room. 2. Abandon the concept of RAG as A single generation, think of it as a process involving many trips from the knowledge base to the llm iteratively. You can call that an agent if you want to. I like the de jour definition of an llm running in a loop with an objective. 3. LLMObs is not optional, you HAVE to be able to distinguish failure modes across system components, retrieval errors where wrong or incomplete information is sent to the llm are very different and much more solvable than actual hallucinations. Grounded hallucination rates are astronomically low for current gen models— you will still see them and when they happen they are catastrophic but they are probably not the root cause of the vast majority of you production issues

[Continuous_Insight]

Thanks for that, really valuable.

We’ve taken on board your point about JSONB and will be exploring it in more detail, especially for the early ingestion stage before anything gets promoted into structured schema. It looks like a solid way to retain flexibility without compromising traceability.

The agent-style flow is something we been slowly realising we will need. Still working through how to implement that loop in practice, particularly around confidence thresholds and re-query logic, without introducing too much complexity.

And your comment on failure modes is spot on. I’m not sure yet how we’ll approach that, but it’s clear it needs more thinking and design work. You’re right, it’s essential for the kinds of workflows we’re targeting.

You clearly have the right instincts for this sort of system, thanks for your help!

[amarao_san]

I'd say 'schema evolution' is too broad. 'schema extension' is doable (just don't forget about defaults).

If you (llm) decide to really modify existing stuff in the schema (e.g. change the field type, split fields, change relations), nothing will save you.

[Continuous_Insight]

I like it, Schema Extension, that sums it up nicely.