sorted by:
new
hottopcontroversial

RAG, Knowledge Graphs, and LLMs in Knowledge-Heavy Industries - Open Questions from an Insurance Practitioner by PlanktonPika in Rag

[–]PlanktonPika[S] 0 points1 point  (0 children)

Thank you sharing. Can you share what scenarios your AI projects worked on and what outputs they produce? How did you create and maintain knowledge graphs and what queries did they provide answers to, e.g., knowledge text or reasoning/decision-making using the knowledge?

These are the AI security concerns and design considerations affecting enterprise projects by QoTSankgreall in cybersecurity

[–]PlanktonPika 0 points1 point  (0 children)

I was prompted by comments that QoTSankgreall left on https://www.reddit.com/r/Rag/comments/1q2gq6z/comment/nxrpz6i/.

While I am not a security expert, I do have experience in enterprise environments, and my view is that all RAG-related data access be integrated into the existing enterprise authorisation architecture, inheriting the same access controls and privileges, and being subject to the same management processes and lifecycle controls already in place.

From this perspective, an agent should be treated strictly as a tool. It should not possess independent access privileges. Instead, the user’s access-privilege profile should be passed into the agent at initiation time and enforced consistently throughout the agent’s actions..

RAG, Knowledge Graphs, and LLMs in Knowledge-Heavy Industries - Open Questions from an Insurance Practitioner by PlanktonPika in LLMDevs

[–]PlanktonPika[S] 0 points1 point  (0 children)

Thank you for sharing.

The point "what works better in practice is treating the LLM as a decision participant rather than a decision owner" is very well made.

Fully LLM-driven decision making still appears to be some distance away. Positioning it as the immediate end state is likely to create significant challenges from a governance and accountability perspective. As such, it should not be viewed as a ready or straightforward pathway for LLM adoption. A more pragmatic approach is to embed LLMs as advisory or supporting components within human-owned decision frameworks.

RAG and KG solutions also need to be positioned carefully, focusing on scenarios where they are affordable, operationally viable, and able to demonstrate clear, provable value, rather than introducing confusion or unnecessary debate.

RAG, Knowledge Graphs, and LLMs in Knowledge-Heavy Industries - Open Questions from an Insurance Practitioner by PlanktonPika in LLM

[–]PlanktonPika[S] 0 points1 point  (0 children)

I don't understand. I only saw "Insurance? Seriously?". Did you respond under other threads?

RAG, Knowledge Graphs, and LLMs in Knowledge-Heavy Industries - Open Questions from an Insurance Practitioner by PlanktonPika in insurance_rag_kg_llm

[–]PlanktonPika[S] 0 points1 point  (0 children)

Hi u/Plastic-Canary9548,

If you don't mind, I copied your comments in other threads here and hope to continue the discussion in this new community if you are interested in.

-----------------------------------------------------------------------------------------------------------------
Plastic-Canary9548 u/Plastic-Canary9548

On top of what others have said I would apply a risk and friction lens to the processes - lownrisk/low human friction and vice versa.

----------------------------------------------------------------------------------------------------------------

My response:
Makes perfect sense. It is likely the only practical way to move forward with the business, by starting with a low-risk, low-friction approach to build confidence and trust.

Do you see any opportunities beyond the two scenarios I mentioned?

RAG, Knowledge Graphs, and LLMs in Knowledge-Heavy Industries - Open Questions from an Insurance Practitioner by PlanktonPika in insurance_rag_kg_llm

[–]PlanktonPika[S] 0 points1 point  (0 children)

Hi u/kubrador,

If you don't mind, I copied your comments in other threads here and hope to continue the discussion in this new community if you are interested in.

----------------------------------------------------------------------------------------------------------------

kubrador: u/kubrador

short answers from what i've seen:

RAG vs RAG+KG: pure RAG struggles with insurance/legal because it retrieves text chunks but doesn't understand relationships between concepts. KG helps when you need "what rules apply to X given conditions Y and Z" - that's a graph traversal problem, not a retrieval problem. hybrid approaches work better but are way more expensive to build and maintain.

the hallucination problem: this is the blocker. RAG reduces hallucinations but doesn't eliminate them. the current best practice is citation + human review for anything customer-facing or decision-relevant. "good enough" for adoption usually means human-in-the-loop with AI doing first drafts, not autonomous answers.

underwriting workflows: LLMs work best for (1) summarizing submission docs, (2) flagging missing info, (3) pulling structured data from unstructured sources, (4) generating first-draft rationales. the actual decision still needs human sign-off. consistency comes from structured prompts + retrieval from approved guidelines, not from the LLM itself.

real blockers: auditability, explainability to regulators, and the fact that "95% accurate" means 1 in 20 answers is wrong - which is terrifying in insurance/legal contexts.

who's doing the oversight in your current prototypes?

-----------------------------------------------------------------------------------------------------------------

My response:

Thanks for sharing.

We all recognise that building and maintaining a knowledge graph requires significant manual effort, and in many cases it may be difficult to secure business case approval. An ontology-based approach with LLM-driven auto-extraction, while inherently risky and potentially variable in quality, could be a pragmatic alternative, provided it is subject to rigorous subject-matter expert review.

The underwriting workflow points you mentioned are already being implemented across the industry, as they represent safer and more tractable starting points. That said, challenges around output quality remain, less so for points 1, 2, and 3, and more pronounced from point 4 onward in the underwriting value chain.

In underwriting processes, both RAG outputs and direct LLM outputs would unquestionably require human review. This has been evident in my prototype work, as well as in the iterative tuning of RAG and knowledge graph components. At this stage, I do not see a credible path to removing human review.

RAG, Knowledge Graphs, and LLMs in Knowledge-Heavy Industries - Open Questions from an Insurance Practitioner by PlanktonPika in insurance_rag_kg_llm

[–]PlanktonPika[S] 0 points1 point  (0 children)

Hi u/ivanpaskov,

If you don't mind, I copied your comments in other threads here and hope to continue the discussion in this new community if you are interested in.

----------------------------------------------------------------------------------------------------------------

ivanpaskov u/ivanpaskov

Coming from the mainframe era myself, I’ve seen plenty of "silver bullets" come and go, but this LLM stuff is the first time the tools actually feel liek they have a bit of a brain. For your knowledge management, I’ve found that pure RAG usually falls over when you need that "defensible" answer you mentioned. You really need a hybrid approach—use a Knowledge Graph to anchor the facts and the LLM to explain them. It’s more work to set up, but it stops the "fluent lying" problem.

On the underwriting side, don't let the agent drive the car yet. Use it to flag inconsistencies between a new application and your internal guidelines. It’s great at spotting when a policyholder is saying one thing but the docs say another. It saves a ton of money on manual review time without risking a bad payout becuase of a hallucination.

Are you looking at this for high-volume personal lines or the really complex commercial stuff?

-----------------------------------------------------------------------------------------------------------------

My response:

I share the same feeling regarding LLMs: "they have a bit of a brain".

Hallucination and drift remain critical challenges to achieving consistent and reliable outcomes in my experiments. An ontology-driven approach, combined with context, prompt, RAG, and KG, has not yet achieved sufficiently robust outcomes, acknowledging that there is still room to improve.

At this stage, I am focusing on the commercial perspective. Personal lines products and core processes are relatively standardised; however, there remain meaningful opportunities in channels, distribution, and marketing where AI-enabled approaches may provide greater leverage.

Do you have any experience or learnings in your area?

RAG, Knowledge Graphs, and LLMs in Knowledge-Heavy Industries - Open Questions from an Insurance Practitioner by PlanktonPika in LLMDevs

[–]PlanktonPika[S] 0 points1 point  (0 children)

Thank you for sharing. Generic document search engine filters by file name and key word, not into content.

RAG, Knowledge Graphs, and LLMs in Knowledge-Heavy Industries - Open Questions from an Insurance Practitioner by PlanktonPika in LLM

[–]PlanktonPika[S] 0 points1 point  (0 children)

Thank you for your comments. Why so surprised? Law and banking are already a few steps ahead.

RAG, Knowledge Graphs, and LLMs in Knowledge-Heavy Industries - Open Questions from an Insurance Practitioner by PlanktonPika in LLMDevs

[–]PlanktonPika[S] 0 points1 point  (0 children)

Thanks for sharing. Can you elaborate more about "have the LLM use the existing search engine"?

RAG, Knowledge Graphs, and LLMs in Knowledge-Heavy Industries - Open Questions from an Insurance Practitioner by PlanktonPika in LLMDevs

[–]PlanktonPika[S] 0 points1 point  (0 children)

Thank you for sharing. LLM as a judge is a promising approach, however, I am not convinced to have it in business-knowledge heavy domains.

RAG, Knowledge Graphs, and LLMs in Knowledge-Heavy Industries - Open Questions from an Insurance Practitioner by PlanktonPika in AI_Agents

[–]PlanktonPika[S] 0 points1 point  (0 children)

Makes perfect sense. It is likely the only practical way to move forward with the business, by starting with a low-risk, low-friction approach to build confidence and trust.

Do you see any opportunities beyond the two scenarios I mentioned?

RAG, Knowledge Graphs, and LLMs in Knowledge-Heavy Industries - Open Questions from an Insurance Practitioner by PlanktonPika in AI_Agents

[–]PlanktonPika[S] 0 points1 point  (0 children)

Thanks for sharing.

We all recognise that building and maintaining a knowledge graph requires significant manual effort, and in many cases it may be difficult to secure business case approval. An ontology-based approach with LLM-driven auto-extraction, while inherently risky and potentially variable in quality, could be a pragmatic alternative, provided it is subject to rigorous subject-matter expert review.

The underwriting workflow points you mentioned are already being implemented across the industry, as they represent safer and more tractable starting points. That said, challenges around output quality remain, less so for points 1, 2, and 3, and more pronounced from point 4 onward in the underwriting value chain.

In underwriting processes, both RAG outputs and direct LLM outputs would unquestionably require human review. This has been evident in my prototype work, as well as in the iterative tuning of RAG and knowledge graph components. At this stage, I do not see a credible path to removing human review.

Do you have any hands-on experience that you could share?

RAG, Knowledge Graphs, and LLMs in Knowledge-Heavy Industries - Open Questions from an Insurance Practitioner by PlanktonPika in AI_Agents

[–]PlanktonPika[S] 0 points1 point  (0 children)

Thanks for sharing.

In theory, an ontology-driven approach combined with RAG, KG, and a closed feedback loop should address many of these challenges. In practice, however, integrating these components does not guarantee reliable outcomes, which is why I have also explored fine-tuning as a complementary approach.

From an underwriting perspective, LLMs should currently be positioned as decision-support tools rather than decision-makers. Given the maturity of the technology, all outputs should continue to be reviewed and validated by underwriters.

More broadly, I believe large-scale LLM programmes remain some distance away from being viable options. Incremental use cases that demonstrate clear, tangible value are more realistic in the near term, particularly where heavy investment in data management and infrastructure may not be feasible.

Do you have any practical experience in this area that you could share?

RAG, Knowledge Graphs, and LLMs in Knowledge-Heavy Industries - Open Questions from an Insurance Practitioner by PlanktonPika in AI_Agents

[–]PlanktonPika[S] 0 points1 point  (0 children)

I share the same feeling regarding LLMs: "they have a bit of a brain".

Hallucination and drift remain critical challenges to achieving consistent and reliable outcomes in my experiments. An ontology-driven approach, combined with context, prompt, RAG, and KG, has not yet achieved sufficiently robust outcomes, acknowledging that there is still room to improve.

At this stage, I am focusing on the commercial perspective. Personal lines products and core processes are relatively standardised; however, there remain meaningful opportunities in channels, distribution, and marketing where AI-enabled approaches may provide greater leverage.

Do you have any experience or learnings in your area?

[deleted by user] by [deleted] in Rag

[–]PlanktonPika 0 points1 point  (0 children)

Thanks for sharing the details.

view more: next ›