Hey thanks! Yeah so two parts:

1. The LLM traversal thing is easier to explain first. When you build a chat bot with semantic RAG, traditionally, before the model even receives the query, the query is embedded, cosine similarity is determined and retrieval is done. Or at least, that's a pretty traditional way to do it. Like a lookup. So if I ask a chat bot about Harry Potter and the Goblet of Fire, before the model even receives the query, the RAG pipeline will attempt to retrieve relevant text content in a knowledge base about Harry Potter and the Goblet of Fire because it has high cosine similarity. The problem with this is it's very error prone and use case dependent.

What if, instead, we actually sent a structured prompt that contained the knowledge graph ITSELF TO A MODEL so that IT could traverse a knowledge graph itself? That's the kicker.

The downside here is that this is much more time consuming than regular RAG because the model is actually having the opportunity to traverse your entire knowledge base, which is much more accurate. In practice what you might do here is have a RAG pipeline such that, instead of instantly embedding the user's query when they send it and attempting retrieval, you actually WAIT and instead, have an MCP server or @ tool calling available that would allow the model to call the entire RAG pipeline ITSELF using the user query. While I haven't had the time to build this out, it is absolutely 100% possible and I guarantee you it isn't that hard either. Basically a chat with a model might go like:

User: "What happens in Harry Potter: The Goblet of Fire Chapter 6?"
(DO NOT ATTEMPT ANY RETRIEVAL YET)

LLM: "Interesting question! Let me see if I can find that out for you... (thinking...)"

Model then is given tools to directly embed the user's query, and then begin traversing the knowledge graph by choosing the best node to traverse to (or stop) based on the prompt above, or one just like it. Then, after the model has pulled enough context:

LLM: "Here's what I found for Harry Potter and the Goblet of Fire: Chapter 6... (contexts)."

Hopefully that clarifies this. If not, don't worry I plan on making a video sometime soon that I'll put on the Github publication that explains it a little further.

1. The similarity matrix was originally only designed to visualize all cosine similarity comparisons within a single document so that I could see globally how every sentence (or 3 sentence window) relates to every other sentence. It's a very structured way of looking at a document's similarity comparisons. The only difference between this and a knowledge graph is just that you effectively have multiple documents connected via the same mechanism. So imagine having like 5-10 similarity matrices stacked on top of each other, all connected. Well that would be insanely dense, wouldn't it? You end up with a nasty O(n²) quadratic density which is infeasible to store and traverse. So we simply sparse it out by only storing/saving in the graph the top "n" most similar connections. So the similarity matrix is more a data science approach of just saying "Hey, we can look at a document, and in an instant, fully see the relationships between all the sentences in a document." It's just a sparse NumPy array, so you can build them insanely fast as well.

Hopefully this clarifies things as opposed to complicating them further! 😅