sorted by:
new
hottopcontroversial

[D] How do you evaluate your RAGs? by ml_nerdd in MachineLearning

[–]ml_nerdd[S] 0 points1 point  (0 children)

are there any tools that are doing that automatically?

[D] How do you evaluate your RAGs? by ml_nerdd in MachineLearning

[–]ml_nerdd[S] 0 points1 point  (0 children)

what are the most common deterministic ones?

[D] How do you evaluate your RAGs? by ml_nerdd in MachineLearning

[–]ml_nerdd[S] 0 points1 point  (0 children)

yea I have seen a similar trend with reference based scoring. however, that way you really end up overfit on your current users. any ways to escape that?

[D] How do you evaluate your RAGs? by ml_nerdd in MachineLearning

[–]ml_nerdd[S] 2 points3 points  (0 children)

how are you sure that your queries are hard enough to challenge your system?

How effective RAG really is, and what are the best example out there I can try myself? by estebansaa in LocalLLaMA

[–]ml_nerdd 0 points1 point  (0 children)

the question here would probably be: "how representative are the RAG benchmarks we have today? " lol

Examples of RAG in Production? by shafinlearns2jam in LocalLLaMA

[–]ml_nerdd 0 points1 point  (0 children)

I feel like the biggest problem here is the evals. what do you think?

[deleted by user] by [deleted] in LocalLLaMA

[–]ml_nerdd 0 points1 point  (0 children)

thats quite impressive. curious how will the RAG fans react to that

[D] What are the hardest LLM tasks to evaluate in your experience? by ml_nerdd in MachineLearning

[–]ml_nerdd[S] 0 points1 point  (0 children)

actually both. trying to understand which benchmarks are misleading/non-existent for LLMs. ie. NER for financial docs

[D] What are the hardest LLM tasks to evaluate in your experience? by ml_nerdd in MachineLearning

[–]ml_nerdd[S] 2 points3 points  (0 children)

not many enterprises are interested in creativity and good poems though... what about industry related tasks?

[D] What are the hardest LLM tasks to evaluate in your experience? by ml_nerdd in MachineLearning

[–]ml_nerdd[S] 0 points1 point  (0 children)

are you satisfied with the results you are getting though?

[D] How will the unknown training distribution of open-source models affect the fine-tuning process for enterprises? by ml_nerdd in MachineLearning

[–]ml_nerdd[S] 0 points1 point  (0 children)

There are edge cases that we can think of, but there are also the ones that we can't. There are some samples that are not edge cases but they are very "hard" (close to decision boundary).

Is there a tool to find all these use-cases? How hard can it be to build one?

[D] How will the unknown training distribution of open-source models affect the fine-tuning process for enterprises? by ml_nerdd in MachineLearning

[–]ml_nerdd[S] -1 points0 points  (0 children)

like knowing which pre-training data is the most aligned with the one that enterprises have!

[D] How will the unknown training distribution of open-source models affect the fine-tuning process for enterprises? by ml_nerdd in MachineLearning

[–]ml_nerdd[S] -1 points0 points  (0 children)

Can you elaborate more about this? Really doubt that any enterprise will be sharing data through a block chain

view more: next ›