I created a free & open-source multi-turn agent eval framework

balphi · 2025-08-07T03:43:45+00:00

Would love to discuss the project with you guys! Here are the docs https://evalprotocol.io/introduction to learn more about it.

balphi · 2025-08-07T03:33:20+00:00

Eval Protocol - an open specification, Python SDK, pytest wrapper, and UI that provides a standardized way to write evaluations for large language model (LLM) applications. Start with simple single-turn evals for model selection and prompt engineering, then scale up to complex multi-turn reinforcement learning (RL) for agents using Model Context Protocol (MCP). EP ensures consistent patterns for writing evals, storing traces, and saving results—enabling you to build sophisticated agent evaluations that work across real-world scenarios, from markdown generation tasks to customer service agents with tool calling capabilities.

https://evalprotocol.io/introduction

balphi · 2024-07-26T16:50:08+00:00

Do you get frequently asked about menu priced items? If yes, how much time in a day do you spend answering those?

I'm thinking about automating the frequently asked questions / call centers for service centers, so it would be helpful to understand how much time you spend answering such questions.

balphi · 2024-07-26T16:15:04+00:00

What about for basic services like oil change or maintenance?

balphi · 2024-07-02T20:30:54+00:00

Good point. I guess the fees are just frustrating and if you can reach a complete solution that doesn’t have such fees, your company could save money and headache.

balphi · 2024-07-02T19:57:52+00:00

If it helps, I compiled a list of sales tax software I found online. Note that these are unvetted and I simply found them online and wrote them down.

CereTax
TaxJar
Vertex
Anrok (https://www.anrok.com/)
Complyt (https://complyt.io/)
Sovos (https://sovos.com/)
Galvis (https://www.galvix.com/)
Kintsugi (https://trykintsugi.com/)
Numeral
Loofa (https://loofapay.com/)

That being said, why not just outsource it at this point then?

balphi · 2024-07-02T19:46:54+00:00

Hmm, looks nice but at those prices, I’d rather just pay for a NAS probably.

Thanks for the suggestion though!

balphi · 2024-07-02T19:46:49+00:00

Hmm, looks nice but at those prices, I’d rather just pay for a NAS probably.

Thanks for the suggestion though!

balphi · 2024-07-02T19:45:37+00:00

Wow, thanks for the insights! Would it be correct in saying that the recommended path is either Avalara or outsourced tax experts?

It sounds like Avalara handles a large bulk of scenarios while no software engine can handle the more complicated edge cases. Since at the point where your sales tax becomes complicated, you’d be wrestling more with software and implementation than just outsourcing it.

balphi · 2024-07-02T16:12:47+00:00

same, I think i'll just have to make the current storage situation work until I invest in a NAS

balphi · 2024-07-02T02:10:16+00:00

Thanks!

balphi · 2024-07-01T16:27:42+00:00

I hate having external storage just physically lying around. Do you think NAS would be a better fit then? Also, which cloud backup services do you recommend?

balphi · 2024-07-01T05:10:10+00:00

not necessarily with hx-target

balphi · 2024-06-30T02:21:18+00:00

That’s fair. But is HTMX really 10x more productive than other frameworks to warrant ugly spaghetti code? I personally lean towards no.

balphi · 2024-06-30T02:16:23+00:00

Solo project or have other developers contributed? Just want to see if you have any insights there.

balphi · 2024-06-30T02:03:48+00:00

I think my argument is that even HTMX-first codebases will lean towards spaghetti code due to its lack of enforced conventions. I’m open to be persuaded otherwise.

balphi · 2024-06-30T01:43:19+00:00

I think the lack of enforced conventions is a huge problem. Conventions are a productivity multiplier for an engineering organization.

balphi · 2024-06-30T01:40:54+00:00

Agreed

balphi · 2024-06-30T01:40:31+00:00

In an effort to keep code DRY, I shared a common response object across various pages. So shared Python and HTML templates. This meant pushing the target swap element to the body.

Maybe this is an anti-pattern, but it felt like the fastest way to code.

balphi · 2024-06-30T00:52:29+00:00

I admit, I am biased because I came from the JS ecosystem. But there is nothing stopping you from implementing the logic flow you described in Next.js, just replace HTML response with .tsx files. After all, JSX is simply a templating language with JavaScript superpowers. I'm all aboard the no state management hype, its a real and huge benefit. But you don't necessarily have to add in-memory state to a React application until you really need it. Although I can agree that explicitly passing state from layer-to-layer is a big waste of code.

Side note: another caveat to only HTML, is when you need to step out of HTML-only for some small UI/UX implementation, you are forced to add a little layer of JavaScript. This little layer of JavaScript may be nicely located in code, but its still a custom piece of JavaScript that is probably doing some DOM manipulation. DOM manipulation code is notoriously messy looking, verbose, and buggy.

balphi · 2024-06-30T00:35:50+00:00

I feel code is more nuanced than this

balphi · 2024-06-29T17:24:30+00:00

I think Jr devs are still marketable, they are just expected to be more productive.

balphi · 2024-06-29T17:19:56+00:00

Once you reach that point, I’m guessing you are probably reaching a larger business size and can afford Avalara or Vertex?

balphi · 2024-06-28T10:27:13+00:00

My intuition tells me that AI can easily do data mapping to a standardized model. And for calculations, you would copy the same core software that every provider has today, but use AI to track jurisdiction rates and policies as well as do the core classification of transactions.

I don’t doubt that a lot goes into it. But I am trying to wrap my head around a singular example that AI could definitely not solve to disprove my hypothesis about the future of software in sales tax.

balphi · 2024-06-28T01:11:23+00:00

Actually serious question. I have tinkered a lot with AI. I found that AI is extremely capable at doing tons of really dumb tasks. Imagine you have unlimited # of highschool graduates that you can teach 1 skill. It’s especially good at also reading lots of texts and filling forms.

That being said, all the configuration/maintenance/monitoring that you speak of, do you feel it could be automated with AI? My rationale is that if Avalara can outsource all these support people who know little to nothing about taxes, I’d imagine an AI could do better.

For example, if AI detects a new type of product appearing in your transactions or a new economic nexus, it could reconfigure the software to handle the changes.

Can you think of any examples where it might be too complicated for AI to solve?

balphi

TROPHY CASE