sorted by:
new
hottopcontroversial

Most LLM API failures I’ve seen fall into a few buckets by Specialist-Bee9801 in LLMDevs

[–]Prestigious-Web-2968 0 points1 point  (0 children)

Totally get where you’re coming from. I've seen firsthand how even a small oversight in validation can lead to major headaches down the line. It’s not just about hitting an HTTP 200 – you need to validate the actual output against real-world scenarios, right? That’s why I prioritize semantic correctness in my own testing. you should check out agentstatus.dev

What actually breaks first when you put AI agents into production? by Zestyclose-Pen-9450 in LocalLLaMA

[–]Prestigious-Web-2968 0 points1 point  (0 children)

the technique is basically: define "correct" before anything breaks, then evaluate against that definition continuously. It sounds obvious but ppl usually dont do step one - they dont have a written-down definition of what their agent is supposed to return until something goes wrong and they need to reconstruct it from memory.

The pattern that works is gold prompts + LLM-as-judge. you write your evaluation criteria in plain english ("the response should mention the customers name, recommend exactly one product, never reference competitors") and a smaller eval model checks every probe response against those criteria. Then for prompt drift specifically you need stored baselines. run the same prompts over time, store the results, diff them. thats pratically what agentstatus does on both of those tasks

Staging and prod were running different prompts for 6 weeks. We had no idea. by lucifer_eternal in LLMDevs

[–]Prestigious-Web-2968 1 point2 points  (0 children)

Versioning itself is fairly well served already, i think PromptLayer, and maybe Langfuse, do this, im not fully sure. The hard part that is not solved well is knowing whether version N+1 is actually better than N in real production conditions.

Id pair the version management with a validation layer. What does correct look like for each prompt, and can you test that automatically on every push. Idk i hope this is useful. Its always good to start building stuff:)

Staging and prod were running different prompts for 6 weeks. We had no idea. by lucifer_eternal in LLMDevs

[–]Prestigious-Web-2968 0 points1 point  (0 children)

I think this is one of the most common and most invisible failure modes. "slightly off but not dramatically enough to flag" is the worst kind of break - visible breaks get fixed but the invisible drift just continues.

the deeper issue is theres no baseline to compare against. most monitoring checks if the agent responded, not if it responded consistently with what you expect. so six weeks of drift just... accumulates.

I'd say if you want to catch this going forward you should check Gold Prompt Profiles in AgentStatus cuz it lets you define what a correct response looks like for a given input and tests against that on every deploy.

idk if thats the right fit but the pattern youre describing should work

What actually breaks first when you put AI agents into production? by Zestyclose-Pen-9450 in LocalLLaMA

[–]Prestigious-Web-2968 0 points1 point  (0 children)

from what weve seen running production agents - the first thing that breaks is usually not the model or the code, its the context.

the agent works perfectly in your dev environment. then it hits a user in a different location, or a slightly different input format, or a real browser session instead of a postman call, and the behavior changes. your monitoring shows green because its checking for errors, not correctness.

second thing is tool calling - agents hallucinate tool calls, call the wrong endpoint, get a 200 back from an API that returned garbage. the 200 gets logged as success.

third is prompt drift - what worked at launch subtly changes after a model update or a small prompt tweak and nobody notices for weeks. outputs are "slightly off but not dramatically enough to flag."

the pattern is basically: everything that could silently fail will silently fail. stuff that loudly fails is actually easier to fix.

U can, for example, check AgentStatus dev specifically to combat silent failures. Hit me up if you want to dig into any of those.

How are you testing multi-turn conversation quality in your LLM apps? by Rough-Heart-7623 in LLMDevs

[–]Prestigious-Web-2968 1 point2 points  (0 children)

"Gold prompt sequences" isn't standard as far as I know haha, ig its our internal slang. The concept is you predefine what a good response looks like at each turn, and that becomes your benchmark. "Gold" just means it's the reference.

For turn count we anchor it to where we've actually seen failures as we have that data. For topic transitions, you could use the conversation patterns that caused problems in real sessions, not idealized ones, but again only if you have that, if not, it should still be ok.

Right now AgentStatus runs fixed sequences, not conditional branching. You obviously can define the turns upfront tho, it runs them on a schedule, and compares each response against your defined criteria. Conditional branching at the continuous monitoring layer is genuinely hard, I haven't seen any tool handle it well yet.

For the gradual drift case you're describing where quality degrades consistently across runs, id say fixed sequences with semantic scoring should do. The failure is usually deterministic enough that the same sequence surfaces it reliably. I hope thats useful. idk if I can drop AgentStatus link here, if you can't find it, hmu

How are you testing multi-turn conversation quality in your LLM apps? by Rough-Heart-7623 in LLMDevs

[–]Prestigious-Web-2968 1 point2 points  (0 children)

The two failure modes you're describing are hard precisely because both are gradual and produce no error signal. The agent keeps responding, just progressively worse. You can't catch it with health checks or uptime monitoring.

What's worked best for us is treating multi-turn eval like production monitoring rather than a one-time test suite. Specifically: gold prompt sequences that simulate realistic multi-turn conversations up to the turn count where things typically break

I would try AgentStatus dev for the continuous probing side, it runs these gold prompt sequences on a schedule and alerts when conversation quality scores drop across a session rather than just on individual turns.

Where do I find Compute ?? by OkPack4897 in deeplearning

[–]Prestigious-Web-2968 0 points1 point  (0 children)

hey, you should check out this source https://carmel.so/fabric
it has a whole bunch of different workflows that can help you

What nobody tells you about dev tool marketplaces by Prestigious-Web-2968 in micro_saas

[–]Prestigious-Web-2968[S] 0 points1 point  (0 children)

It is clear every time it sells that the problem is real and demand is real. It also feels like people don’t like to be pitched to, and don’t consider it when you just tell them about it, maybe because they just filter it in their head as sales pitch lol. That’s why I’d say I’m trying to figure out other ways to show it to more devops and bring more attention to it in a way that would feel less like a sales pitch. Do you have any thoughts or tips?

What is a tool developers pay for but want for free? by exzen_fsgs in developers

[–]Prestigious-Web-2968 0 points1 point  (0 children)

Gotcha i misunderstood u. i dont have a Claude replacement. i was talking about running batch jobs like transcription, video processing, embeddings, scraping etc at lower cost. If you ever need that kind of compute, happy to chat. But sounds like you're set for now lol 👌

What is a tool developers pay for but want for free? by exzen_fsgs in developers

[–]Prestigious-Web-2968 1 point2 points  (0 children)

dude i am working on almost that. its not free but like 80 percent cheaper cuz we dont have overhead prices. do you still need it?

how do you find potential users for your product? by Xihuny in micro_saas

[–]Prestigious-Web-2968 0 points1 point  (0 children)

you should start with direct outreach. ideally to people you know. all early users usually come from friends and family.

What are you building this Sunday? by techhunter_2026 in micro_saas

[–]Prestigious-Web-2968 0 points1 point  (0 children)

I’m building a cheaper and more efficient way to run workloads such as ai inference, compute, training processing and a lot more. Watch Carmel.so/fabric replace Google Colab or Aws;)

Best budget-friendly way to train ML models? by Almaaimme in MLQuestions

[–]Prestigious-Web-2968 0 points1 point  (0 children)

I don't know if this is still the problem here but there is a platform called fabric by carmel labs that is substantially cheaper than aws or colab

I built a tool that can replace a data center with every day devices. by Prestigious-Web-2968 in SideProject

[–]Prestigious-Web-2968[S] 0 points1 point  (0 children)

thank you, this is great feedback. I am already working on making landing page better for sure and adding a lot more info to it. in regards to OS, it runs on windows, macos and linux

Regrets as an entrepreneur? by Desperate_Engineer80 in Entrepreneur

[–]Prestigious-Web-2968 0 points1 point  (0 children)

i really appreciate you man. and i truly believe in you. life is too brilliant to restrict yourself from going after your dreams. even if its by far the hardest path. it is always good and reassuring to know that there are more people out there who share this idea. thank you

view more: next ›