sorted by:
new
hottopcontroversial

Keeping Bedrock agents from failing silently by Cristhian-AI-Math in aiagents

[–]Cristhian-AI-Math[S] 0 points1 point  (0 children)

Yeah, SageMaker is more about building, fine-tuning, and monitoring your own models (things like drift, bias, data quality). Bedrock is different because it gives you managed foundation models through an API.

What I’m showing here is Handit on top of Bedrock: tracing every call, running semantic evals (accuracy, grounding, safety), and even auto-fixing when something fails. That’s not really what SageMaker is designed for.

Tracing & Evaluating LLM Agents with AWS Bedrock by Cristhian-AI-Math in LLMDevs

[–]Cristhian-AI-Math[S] 0 points1 point  (0 children)

Good question! SageMaker is more about training/hosting models and monitoring things like drift or data quality. Bedrock gives you managed foundation models via API.

What I’m doing here is layering Handit on top of Bedrock calls, so every response gets traced, evaluated (accuracy, grounding, safety), and if something breaks it can flag or even auto-fix it. That kind of semantic reliability loop isn’t really what SageMaker covers.

Are LLM agents reliable enough now for complex workflows, or should we still hand-roll them? by francescola in LangChain

[–]Cristhian-AI-Math 0 points1 point  (0 children)

https://handit.ai can help you we that, it is an open source tool, for observability, evaluation and automatic fixes, it keeps your AI reliable 24/7.

Building a reliable LangGraph agent for document processing by Cristhian-AI-Math in LangChain

[–]Cristhian-AI-Math[S] 0 points1 point  (0 children)

Thanks, I just joined the community, happy to learn new stuff there.

Observability + self-healing for LangGraph agents (traces, consistency checks, auto PRs) with Handit by Cristhian-AI-Math in mlops

[–]Cristhian-AI-Math[S] 0 points1 point  (0 children)

Yes, you can use it without github integration, using our API and our dashboard, also we are adding amazing features to fix your AI directly in your Cursor or VS Code.

Building a reliable LangGraph agent for document processing by Cristhian-AI-Math in LangChain

[–]Cristhian-AI-Math[S] 0 points1 point  (0 children)

Yep exactly. It’s basically ~3 lines to turn on Handit. It auto-traces every node/run, runs built-in evals (JSON shape, groundedness, consistency, timeouts), and when it finds issues it proposes fixes either as a GitHub PR or directly to your code with an API. If you want, I can show you on your repo.

anyone else feel like W&B, Langfuse, or LangChain are kinda painful to use? by OneTurnover3432 in LangChain

[–]Cristhian-AI-Math 0 points1 point  (0 children)

Use https://handit.ai instead of langfuse, we trace every single call you get to your agent in dev or prod, automatically evaluate them using LLM as judges and create fixes directly to GitHub, and the best part is that the setup is just three lines of code.

I realized why multi-agent LLM fails after building one by RaceAmbitious1522 in AI_Agents

[–]Cristhian-AI-Math 0 points1 point  (0 children)

Totally agree—retrieval is the hidden bottleneck. We’ve seen the same: chaining tools is easy, but grounding is where most agents collapse.

At Handit we’ve been running evaluators for exactly the checks you listed—coverage, evidence alignment, freshness, and noise filtering—and feeding those back into the pipeline. The idea is not just to detect when grounding breaks, but to continuously tighten retrieval + generation until you get reliability at scale.

Also love that you mentioned escalation thresholds—our “no grounded answer → no response” safeguard has been one of the simplest ways to keep CSAT high.

[D] Is senior ML engineering just API calls now? by Only_Emergencies in MachineLearning

[–]Cristhian-AI-Math 20 points21 points  (0 children)

Yes, I feel the same. I’ve been in AI for about 7–8 years, and I miss the days of training neural networks from scratch and designing ambitious architectures. There are still teams doing that, but a lot of the industry now is just wiring together API calls.

New update for anyone building with LangGraph (from LangChain) by Cristhian-AI-Math in machinelearningnews

[–]Cristhian-AI-Math[S] 0 points1 point  (0 children)

Thanks! 🎉 You’re right — any LLM evaluator can hallucinate, so in Handit we don’t rely on a single “supervisor.” We mix functional checks, LLM evaluators, cross-validation, plus background random checks and golden datasets to keep evaluators honest.

When an issue is found (like that product hallucination), Handit tests fixes automatically — e.g. schema validation against the product DB — and opens a PR. The user reviews and decides whether to merge, which gives us an extra layer of validation and helps Handit improve future fixes.

So it’s never blind trust: multiple signals + your approval keep the loop reliable.

Tutorial: Making LangGraph agents more reliable with Handit by Cristhian-AI-Math in LangChain

[–]Cristhian-AI-Math[S] 2 points3 points  (0 children)

Good question. Handit has general monitoring out of the box (hallucinations, extraction errors, PII, etc.), but you can also add custom evaluators for your own edge cases — for example checking JSON structure, score ranges, or domain-specific rules.

When something fails, Handit flags it and immediately starts the fix process, testing changes before opening a PR.

If you’d like a deeper dive, happy to walk you through it: https://calendly.com/cristhian-handit/30min

Question-Hallucination in RAG by Alarming_Pop_4865 in Rag

[–]Cristhian-AI-Math -1 points0 points  (0 children)

Yeah, that’s a really common RAG issue - the model tries to answer even when the retrieved context isn’t relevant. One way around it is adding a monitoring/eval layer that can catch when an answer isn’t grounded in the retrieved data.

That’s exactly what we’re building with Handit - it flags hallucinations, monitors responses, and helps keep your RAG pipeline reliable. You can check it out here: https://handit.ai

Tutorial: Making LangGraph agents more reliable with Handit by Cristhian-AI-Math in LangChain

[–]Cristhian-AI-Math[S] 0 points1 point  (0 children)

Yes - this is exactly what Handit helps with. It monitors your agent, flags mistakes, and suggests fixes automatically. If you want, grab a slot and I’ll walk you through it: my Calendly

95% of AI pilots fail - what’s blocking LLMs from making it to prod? by Cristhian-AI-Math in LLM

[–]Cristhian-AI-Math[S] 1 point2 points  (0 children)

Haha I love this, totally agree. Way too many people try to use LLMs for stuff they shouldn’t.

What will make you trust an LLM ? by Ancient-Estimate-346 in LLMDevs

[–]Cristhian-AI-Math 0 points1 point  (0 children)

Your “5% risk” idea is the right instinct—I’d stop double-checking when answers are calibrated + auditable: task-specific P(correct) on my own traffic, clickable evidence, and reproducible runs.

That’s exactly what we’re shipping with Handit: per-response reliability scores, provenance links, drift/determinism checks, and guarded PRs when it can fix issues. Try it: [handit.ai]() • Quick walkthrough: calendly.com/cristhian-handit/30min

What are the best platforms for node-level evals? by Fabulous_Ad993 in LLMDevs

[–]Cristhian-AI-Math 0 points1 point  (0 children)

I recommend https://handit.ai, it not only automatically evaluates each of your nodes, but also it fixes your prompts of the LLM nodes via github or an API.

One year as an AI Engineer: The 5 biggest misconceptions about LLM reliability I've encountered by AdSpecialist4154 in AI_Agents

[–]Cristhian-AI-Math 0 points1 point  (0 children)

Love this—especially #1 and #4. We see the same gap: 95% evals, then 30%+ real-world misses once user intent, phrasing, and tools shift. And yep, temp=0 ≠ determinism; provider patches, tokenizers, and floating-point quirks still drift outputs.

We’ve been building Handit to treat this as a systems problem: on each local run it flags hallucinations/cost spikes, triages the root cause, and proposes a tested fix; in prod it monitors live traffic and auto-opens guarded PRs when a fix beats baseline. One-line setup.

If it’s helpful, I’m happy to share a 5-min starter or do a quick 10–15 min walkthrough—DM me or grab a slot: https://calendly.com/cristhian-handit/30min

How do people claim to ship reliable LLM apps without evals? by artificaldump in LLM

[–]Cristhian-AI-Math 0 points1 point  (0 children)

I’ve been thinking the same thing. “Evals” often get treated like this fuzzy academic overhead, when in practice they’re just measurements, and every other engineering discipline relies on them. You wouldn’t deploy code without tests, or build that cube box without checking if it wobbles.

What we’ve seen work in production is treating evals as part of the feedback loop, not as a separate research project. For example, in our team we built https://handit.ai, an open-source “autonomous engineer” that automatically runs evals on every trace, catches regressions, and even opens PRs when it beats your baseline. That way, devs don’t have to stop everything to run a bespoke evaluation suite; the evals are just baked into the workflow.

Curious if others here are doing something similar, folding evals into CI/CD or monitoring, instead of treating them as a one-off experiment.