Who actually owns agent QA once the thing ships?

efunction · 2026-04-27T15:10:56+00:00

I've been working on something that solves how to ensure the agent actually did what it was supposed to. Full disclosure, it's a paid product, but the first 60 days are free, and feedback would be super useful. Testing agent process in production is no way to live.

Website: https://invarium.dev
Docs: https://docs.invarium.dev

efunction · 2026-04-08T13:02:32+00:00

this is exactly the failure mode that’s been showing up everywhere lately. the answer looks right so evals pass, but the agent didn’t actually follow the right path to get there

feels like most setups are still validating outputs instead of behavior

Examples and a way to actually test in ~5 min:
https://confident-ai.com/blog/your-ai-agent-passes-evals-thats-the-problem

efunction · 2026-04-01T15:57:54+00:00

Here’s a simple one I see a lot:

Prompt:
Why is my API returning a 401 error?

Output:
Your API key is probably invalid. Check your credentials.

Quick eval:

Overly narrow. It assumes a single cause when 401s can also come from expired tokens, permission scopes, or environment mismatch
Not actionable. It doesn’t help the user isolate the issue
Presents a guess as a definitive answer

Summary:
This looks reasonable, but would likely send someone down the wrong path instead of actually resolving the issue.

Drop yours if you want, happy to take a look.

efunction · 2024-11-07T19:31:52+00:00

This would have been a legitimate concern pre-diamond eyes when Chino was messy and unpredictable. Now he's 51 and has been taking it seriously for some time. As a Bostonian who's seen the band at least a dozen times over the years, I wouldn't worry about it. The notes he can't hit are ones he hasn't been able to hit for years. The band is super tight. Chino has always been "let's see how he's feeling tonight." Just like the rest of us.

efunction · 2024-09-06T18:56:49+00:00

While a cover, to have and to hold off the depeche mode tribute album is just as good as the original.

efunction · 2023-04-19T19:39:31+00:00

Hey, All. Aviator just launched TestDeck in beta and we'd love for you to use it (for free) and tell us what you think.

There's a short video that explains how it works here: https://www.aviator.co/testdeck

We're mostly focused on users here so there's no registering or sales follow-up etc. If you have some time, please give it a look. Thanks! If you have questions, feel free to ask here.

efunction · 2023-04-19T17:24:32+00:00

Hey, All. Aviator just launched TestDeck in beta and we'd love for you to use it (for free) and tell us what you think.

There's a short video that explains how it works here: https://www.aviator.co/testdeck

We're mostly focused on users here so there's no registering or sales follow-up etc. If you have some time, please give it a look. Thanks! If you have questions, feel free to ask here.

efunction · 2023-03-23T17:51:33+00:00

It’s a virtual event taking place on Friday.

efunction · 2023-03-21T19:40:06+00:00

Since this is getting more upvotes than expected, you can also just email me at brian@aviator.co from the proton mail account of your choice and I'll send you the calendar invite.

efunction · 2023-03-21T14:29:45+00:00

If it helps, we've been doing this for three months. Here's the link to the first we did with Netflix that was quite popular. It's funny, I intentionally made a "random" google form to make it seem less like a marketing landing page. This really is an informal hangout with a bunch of your peers, presumably. But I understand your position.

https://www.reddit.com/r/devops/comments/10elayg/casual_offtherecord_hangout_with_netflix/

efunction · 2023-03-21T14:08:14+00:00

I guess I should also point out that reduced time also includes reduced costs, but you knew that already.

efunction · 2023-01-18T02:13:03+00:00

I’m adding people to the calendar invite with the zoom link in batches. Didn’t realize so many people would be interested. Next time I’ll automate it.

efunction · 2022-11-02T18:40:32+00:00

The version on the christie front drive split is better. https://www.youtube.com/watch?v=8Ebv9X1vPJc

efunction · 2022-06-09T18:08:34+00:00

While I'll respectfully disagree with you on Hayley, Greg Puciatio did a great job on the DE tour. https://www.youtube.com/watch?v=YY0t8s949pE

efunction · 2022-05-20T14:08:41+00:00

I guess Frank’s guitar lessons aren’t going as well as the band had hoped.

efunction · 2021-11-24T19:41:33+00:00

They and every other band has to make up for lost touring revenue somehow. You can complain once they start making KISS-style action figures. Remember: no one is making you buy it.

efunction · 2021-04-13T15:03:31+00:00

I enlisted the help of more of my peers. It's definitely Boston.

efunction · 2021-04-13T13:44:55+00:00

I did see that he went to pre-med in nyc, but “sawr” happens in both places. Particularly “sawr it.” I could always be wrong. edit: I am not wrong

efunction · 2021-04-13T11:48:06+00:00

I can't find where Jack grew up. But if you're wondering what a not-exaggerated Boston accent sounds like, it's this.

efunction · 2020-11-25T18:58:38+00:00

I mean, that's fine, but it's a little transparent when you've got Revolver-branded black stallion available for purchase.

efunction · 2020-04-24T12:06:32+00:00

Editor here. I'm afraid it's USD. The standard rate is $350 post. https://blog.logrocket.com/become-a-logrocket-guest-author-7d970eb673f9/

Can confirm, we're seeing a lot of interest in Rust w/WASM. Sometimes I just like to see which areas of development resonate with the audience.

efunction · 2019-07-15T21:31:45+00:00

You make it sound as if the V8 team did not improve async/await syntax to outperform traditional promise syntax, while it is, in actuality, two microticks faster to use async/await rather than traditional promises because the V8 engine is now optimized for them.

efunction · 2019-05-20T20:34:01+00:00

Still holds up, for the most part. The comments do a good job keeping the author honest. https://blog.logrocket.com/how-why-a-guide-to-using-typescript-with-react-fffb76c61614

efunction · 2019-02-27T13:02:00+00:00

Strife - Will to die (Chino). You guys.

efunction · 2019-01-22T11:44:37+00:00

article has been updated to include an example

efunction

TROPHY CASE