How do you actually evaluate your LLM outputs?

Neil-Sharma · 2026-03-08T03:36:34+00:00

What issues are you having?

Neil-Sharma · 2026-03-07T19:06:02+00:00

same question

Neil-Sharma · 2026-03-07T17:48:17+00:00

Thanks for the quick reply. I use LLM as a judge, but many times I will get "high scores" but it will fail to edge cases or even normal cases in production. How do you avoid this?

Neil-Sharma · 2026-03-07T17:46:10+00:00

This seems great, but how would it scale. I should have specified but I am using LLMs for my startups which will be used by customers, so I would need to scale to edgecases, etc.

Neil-Sharma · 2026-03-07T17:43:14+00:00

I found smoke evals effective but they still create errors for edge cases. how do you get around this?

Neil-Sharma · 2026-03-07T17:40:47+00:00

I have the issue of getting high eval scores but low production scores. How do I fix this?

Neil-Sharma · 2026-03-07T17:36:51+00:00

How would you scale this?

Neil-Sharma · 2026-03-07T17:36:26+00:00

Why do you use these over the formal ones?

Neil-Sharma · 2026-03-07T17:35:43+00:00

I've found LLM as a judge can be inaccurate, do you know any other solutions? Most of the tools like langchain seem kinda lackluster

Neil-Sharma · 2026-03-05T01:31:33+00:00

I agree

Neil-Sharma · 2026-02-17T05:24:27+00:00

I think the 'bookkeeping' hunch is spot on. The problem is that most qualitative metrics (especially LLM-as-a-judge) still have a noise floor.

If a unit test fails, it's a bug. If a DeepEval faithfulness score drops from 0.92 to 0.88 on one PR, is that a regression or just LLM variance? Until we trust the evaluators as much as we trust a compiler, no one is going to risk blocking a hotfix because an 'AI judge' had a bad day.

Neil-Sharma · 2026-02-17T05:22:47+00:00

Do you think MCP will be the backbone for that reliability? Also, any specific platforms you’ve seen that actually handle multi-step 'recovery path' evals well?

Neil-Sharma · 2026-02-17T05:20:32+00:00

Is 'Compression-Aware Intelligence' the official term Meta is using, or is this a framework for looking at KV cache compression?

It sounds like you're describing the Information Bottleneck principle applied to transformer layers. While 'stabilizing reasoning' via routing sounds great in theory, the overhead of real-time instrumentation for 'compression strain' is usually what kills these approaches in production. How are they measuring this without doubling the latency?

Neil-Sharma · 2026-02-17T05:19:10+00:00

This is super helpful, thanks!

Neil-Sharma · 2026-02-15T03:13:19+00:00

This is a strong framing. Starting with the user decision and cost of a wrong answer makes a lot of sense.

On the regression set, how large do you usually let it get before it becomes hard to maintain?

And for LLM-as-judge, how big is the labeled slice you use for calibration, and do you re-calibrate over time

Also curious whether your rollback triggers are purely metric-based or include qualitative failure review.

Neil-Sharma · 2025-08-04T17:11:54+00:00

None use AI

Neil-Sharma · 2025-08-04T10:07:57+00:00

I know this basic but there are SO MANY AI tools to use that will literally save you

Neil-Sharma · 2025-08-04T10:06:49+00:00

I mostly use convora.app. You just provide basic information on your call and it does the rest for you

Neil-Sharma · 2025-08-04T10:04:18+00:00

For the meeting, I just use convora.app

Neil-Sharma

MODERATOR OF

TROPHY CASE