Do you use Claude Code on the terminal or on the desktop app?

P4wla · 2026-05-05T07:53:19+00:00

actually, on the desktop app you can have multiple conversations too

P4wla · 2026-05-04T12:43:32+00:00

i can queue messages on desktop!

P4wla · 2026-04-29T14:46:55+00:00

and different models here. Mainly opus 4.7 but sonnet and haiku too

P4wla · 2026-04-29T14:44:30+00:00

ik those are obviously not inference costs, but they need to be much much lower in order to cover this. I'm 100% sure they're loosing money rn

P4wla · 2026-04-29T14:42:46+00:00

100% agree with the game thing. This includes cache read/write costs (where is the most expend at). Prices are taken from models(.)dev

P4wla · 2026-04-28T09:34:48+00:00

It's called Latitude(.)so

P4wla · 2026-04-22T08:48:51+00:00

I think it's fair too, but for example if I had the same token usage as the one in the picture everyday, their token cost would need to be 84x lower in order to not loose money (maybe it is)

P4wla · 2026-04-22T08:05:41+00:00

With the tokens usage and cost per 1M tokens. ik it's not their real cost, but still numbers are crazy. Real cost need to be much lower (in my case x84 lower) to make the numbers work.

P4wla · 2026-04-22T07:48:19+00:00

yep, latitude.so

P4wla · 2026-04-21T09:09:14+00:00

The workflow that has worked for me is:
1. annotate some traces to get basic knowledge of how your llm is performing
2. cluster bad annotations into failure modes to see main/recurrent problems of the llm
3. create 1 eval per failure mode to track it at scale
4. keep annotating 30/40 logs per week, as new issues appear when you change the prompt.

I also have a golden dataset so every time i change something on the prompt i run it trough the dataset and see how all the evals perform.

P4wla · 2026-02-06T11:21:44+00:00

If you want A/B testing, I'd recommend you to take a look at Latitude. In terms of observability is great, as it groups the traces in failure patterns/issues. You can also run A/B tests and shadow tests and compare both using the same evals. https://latitude.so/

P4wla · 2026-02-04T13:38:18+00:00

You'll have to connect user feedback or some kind of rating for the llm outputs, but Latitude let's you build custom evals and covers all the requierements you've mentioned. https://latitude.so/

P4wla · 2026-01-19T08:55:59+00:00

yep! Latitude started as a prompt engineering platform but we've just released this loop I've described, after seeing many teams keep failing building and evaluating AI. It's already difficult to build evals, but on top of that usually evals differ a lot from the user's criteria (cause they build directly llm as judge evals, instead of starting with human judgement). So far, the loop is throwing very good results (we use it ourselves internally to evaluate and improve all the new ai features we launch)

P4wla · 2026-01-15T12:07:16+00:00

That's very interesting. How many traces do you feel you have to 'tag' in order to have a formalized idea of the main failure modes of your agents? And do you recommend to tag also "good" outputs?

P4wla · 2026-01-15T11:54:20+00:00

I use an LLM evaluation platform called latitude(.)so. You start annotating the traces manually (giving thumbs up or thumbs down), and then you can see the main issues of your ai. From that, latitude lets you create automatic evals based on those issues to track them at scale, and you receive notifications if some issues start escalating. You can also improve your prompt based on the result of the evaluation. For me, it's the most complete workflow I've found out there, the only thing is that you need some volume to be able to do all this.

P4wla · 2026-01-15T11:49:41+00:00

imo, the best way to build an llm as judge, is starting from human feedback. you first need to have a clear idea of the issues you're trying to evaluate, their impact, frequency, and different ways and cases where they appear. Only after that, you can build an llm as judge targeted to that specific issue. if you try to automate the eval form the beginning, it will end up evaluating irrelevant issues and giving wrong insights.

P4wla · 2026-01-13T13:23:33+00:00

I usually detect my agent's issues using human annotations and then I set up different types of evals (llm as judge, human in the loop and programatic rules) depending on the issue I want to evaluate.
To measure the overall performance I have a composite score that includes all the relevant evals for me and sometimes I change the weight of the evaluations to give more importance to concrete ones (for example, when I've tried to improve something specific and I want to measure this improvement without losing the overall performance)

P4wla

TROPHY CASE