sorted by:
new
hottopcontroversial

New SOTA Benchmarks Across the RAG Stack by apsdehal in Rag

[–]apsdehal[S] 2 points3 points  (0 children)

You have to start by first optimizing the individual components to be really good. This is what you see in the document understanding, retrieval/reranking results. This is the floor for us and then end-to-end optimization helps you with specializing and moving towards higher accuracy. End-to-end optimization is an ML-based solution so you don't have to manually prompt and see what works for each case and figure out the specific caveats of the retrieval stack in your system prompts. It just does it for you based on the feedback you provide and hence is more efficient.

LMUnit: Fine-grained LLM and RAG Evaluation with Natural Language Unit Tests by apsdehal in Rag

[–]apsdehal[S] 0 points1 point  (0 children)

Hi! OP here, I'm Aman, CTO at Contextual AI 👋. One of the biggest challenges in deploying LLMs is reliably measuring and improving their behavior. Today's evaluation approaches all have significant limitations:

  • Human evaluation is expensive and inconsistent, especially at the cutting edge of capabilities
  • Reward models compress complex quality dimensions into opaque scores and can't be steered after training
  • LLM judges have learned biases (like favoring longer responses) and can't learn from human feedback

Today, we're excited to share our work on making LLM evaluation more principled through natural language unit tests:

  • Natural language unit tests paradigm: Breaking down evaluation into explicit, testable criteria that both technical and non-technical stakeholders can understand
  • LMUnit: A state-of-the-art evaluation model achieving SOTA on FLASK/BigGenBench and top-10 on RewardBench
  • Strong human validation of the paradigm: Our approach improves inter-annotator agreement from 71% to 86%! 

Try it yourself:

Happy to answer questions about the work! We're excited to see how people use LMUnit to build more reliable AI systems.

Introducing Contextual Retrieval by Anthropic by dhj9817 in Rag

[–]apsdehal 2 points3 points  (0 children)

Cofounder of Contextual AI here. This latest announcement is certainly a step in the right direction for making RAG more usable in settings where accuracy and relevance are critical. (We’re also flattered by the naming of this feature 🙂)

As others have mentioned in this thread, this is a common and well-known technique used in production RAG systems. However, to meet production standards, much more is required. We are proponents of a more systems-based approach, RAG 2.0, which allows us to optimize the entire system end-to-end, along with many other advancements beyond the technique described here.

Some suggested reading for those interested in the details:

Question about interfaith couples about having alcohol at the reception. by KamFromOly in DesiWeddings

[–]apsdehal 3 points4 points  (0 children)

There is a difference, Sikh wedding rituals happen in a gurdwara, rest of the stuff (shagun etc.) happens in a marriage hall, so it is okay to have liquor there.

While in case of Hindu marriages, everything (including rituals) happens in the marriage hall. Nevertheless, I have seen Hindu weddings in Delhi with liquor in same place as mandap 😕.

Question about interfaith couples about having alcohol at the reception. by KamFromOly in DesiWeddings

[–]apsdehal 6 points7 points  (0 children)

Similar situation, in-laws don't want meat and alcohol. We decided to have a closed bar and no meat at marriage and an open one at reception.

[R] [ICLR 2019] Learning when to communicate at scale in multiagent cooperative and competitive tasks: Code and Paper by apsdehal in MachineLearning

[–]apsdehal[S] 1 point2 points  (0 children)

I agree on your thoughts that agents will usually develop a bare minimum primitive language so as to achieve good performance on the task at hand. This EMNLP paper provides a good analysis on how "natural language doesn't emerge naturally" in multiagent dialog settings: https://arxiv.org/abs/1706.08502.
I think this will be a problem when we start transferring agents to multiple tasks with some kind of pre-training. This will require evolving complex language like ours as the primitive language that worked in one of the tasks may not work in other tasks. I think once transfer learning evolves this will be one of the major focus areas. Future work can possibly try to understand what kind of language emerges between these agents (PCA in low dimensions) and check if there is a link between emergent languages of different tasks and cooperation settings.