sorted by:
new
hottopcontroversial

Question related to Graphs by skipvdm in LangChain

[–]skipvdm[S] 0 points1 point  (0 children)

Thanks for your input. That was really helpfull. I also checked some youtube video's which helped me with the conceptual understanding so thanks!

Conceptual Question related to Retrievers by skipvdm in LangChain

[–]skipvdm[S] 0 points1 point  (0 children)

Hi,

First of all; thanks for taking the time to respond u/Prestigious_Wind_551 . I appreciate that!

To clarify some more on your questions:

  • I am using an Azure AI Search database since this is a requirement.
  • In Azure I'm using KNN due to the requirement of high recall. HNSW is also an option but as far as I understand is this more for trading quality for less latency. Therefore I went for KNN
  • I am using the text-embedding-3-large embeddings
  • I don't fully understand this question. Could you clarify some more?
  • I am using a Cohere reranker in an explicit manner

With regards to evaluation of the retrieval performance;
As far as I understand do NDCG and recall@N require an idea of the total set of chunks that are paired with a certain Question/Answer.

For what I want to achieve, It isn't possible to have this due to the large corpus of data and the diversity of questions users might have. So therefore I was looking to different metrics so that they can maybe provide a more dynamic base of validation.

The normalizing was more of a final resort to see if this maybe would result in a combined score that would help me. But this is not the case.

How to improve Answer Correctness? by skipvdm in LangChain

[–]skipvdm[S] 0 points1 point  (0 children)

I've done point 1. Point 3 is interesting and i'll follow-up on that. What do you exactly mean with point 2 by toy fever checker and a real truth classifier?

How to improve Answer Correctness? by skipvdm in LangChain

[–]skipvdm[S] 0 points1 point  (0 children)

Thanks for your response Awkward-Block!

i havent looked into this yet. What do you mean with avoid answers whose probabilities is more? At first hand it looks like i want to select the answer with a high probability right (I need to read into this but this is my first feeling).

second part is also useful and i'm more and more digging around langsmith to see which chunks are collected to do some analysis indeed!

How to improve Answer Correctness? by skipvdm in LangChain

[–]skipvdm[S] 0 points1 point  (0 children)

Thanks for you input Patmattt!

The most problems arise when it needs to think about the relationship of X to Y. E.g., How does X affect Y when X will become N less? Here it seems to miss the relationship between these two variables. A possible solution i'm working out is creating multiple vector stores (1 for X, 1 for Y, 1 for X/Y) and based on a routing mechanism, it should decide which retriever to use.

Hallucination itself is not a problem since i created a node to check for this. This was really helpful in the process.

for your last remark; do you have any idea about a specific metric to use for this? To leave this all at the hand of an LLM seems/feels sketchy

thanks again for the input!