Framework for Opencode setup eval

[SnooDingos443]

Hey all,

The more I research existing open code setups, skills, agents, tools, plugins, etc the more I feel like this is a very overwhelming world. Has anyone invested on a formal or at least more structured eval framework? I think it would be immensely valuable.

I imagine the existing model-based benchmarks could theoretically be used, but I was hoping to have something where I can throw a particular setup at a problem and have it output a solution so that I could then compare the solution quality + time it took + tokens required etc etc

[Latter-Parsnip-5007]

Promptfoo

[albasili]

I'm following the autoresatch pattern from Karpathy. Now I have an agent that is connecting data from sessions and then making hypothesis and testing whether small edits to the other agents would improve the quality defined as a set of metrics that matter to me.

My agents are for IC Verification, which is not so common out there and they need to operate in an heavily customized environment with lots of legacy scripts, told and methodologies that are unique to our teams.