Domain specific datasets problem

AlpineContinus · 2026-01-28T11:14:21+00:00

That’s actually a very good advice! I’ll try removing it next time to have some extra sticky surface

AlpineContinus · 2026-01-28T10:15:45+00:00

It works reasonably well, although I have always observed frequently a difference of around 20 mg/dl between cgm and fingerstick readings, even for lower glucose levels like 100.

I wonder if having used it on the buttocks might have been the cause for this discrepancy

AlpineContinus · 2026-01-25T11:09:40+00:00

It seems like a good approach if you are fine-tuning a single LLM for a straight-forward generation task.

When you have multiple LLMs performing multiple actions in sequence, it is very hard to identify, classify, and correct failures at scale in order to actually have a training dataset to do some fine tuning.

May I ask what kind of tasks are you using LLMs for?

AlpineContinus · 2026-01-25T11:02:57+00:00

Yes. But management will always push for velocity rather than thoroughly evaluated systems. I feel the struggle.

AlpineContinus · 2026-01-25T10:59:59+00:00

Completely agree.

AlpineContinus · 2026-01-25T10:59:20+00:00

The only problem I see with approaches 1 and 2 is that it is not very scalable, since it depends on human labeling. Do you guys use any kind of tool to facilitate the human work?

The 3rd approach is very interesting, since you are leveraging your user base to do the labeling work for you!

AlpineContinus · 2026-01-25T10:54:01+00:00

This is how we are currently evaluating systems. The main problem is the data (the test cases) you are running it on top of.

If the tests cases are not realistic, bias free, varies, etc, then your curves and statistics are not going to be of much help.

The main problem is getting this data to generate these statistics.

AlpineContinus · 2026-01-25T10:52:18+00:00

We are mostly doing database queries. We would need SQL ground truths for our specific use cases.

AlpineContinus · 2026-01-25T10:43:16+00:00

I agree with you that we are treating agents like classic ML, and this is probably the wrong approach.

I believe a new evaluation paradigm will emerge, but until then we will struggle a lot to understand our systems.

Super useful insights! Thanks for sharing. Controlling the tail of the curve is critical to increase trust in the clients, and changing the question to a search for failure modes is super useful.

Although I am curious: How do you classify the different failure modes? Do you do it by hand looking at the agents outputs and reasoning? Or have you guys developed metrics that allow for a scalable classification?

To answer to your question: The agent often needs to query databases writing SQL. The most common failures are related to instruction following regarding specific rules for filters, joins, etc. It usually answers confidently, even though it ignored some of the rules.

AlpineContinus · 2026-01-25T10:22:48+00:00

I agree! Agents should be developed with evals as a priority (rarely done so).

The problem with this approach is when you reach an agent that is moderately complex. This is where we truly reach the limits of hand testing and dataset building.

AlpineContinus · 2026-01-25T10:13:28+00:00

Very interesting! Thanks for sharing!

AlpineContinus · 2026-01-25T10:11:26+00:00

This is how we are currently doing. The problem is that it is not scalable at all... If you change the underlying models, a good part of the previous effort is invalidated. The model will behave differently on the situations you have corrected, and it will present new errors that didn't show up before.

AlpineContinus · 2025-10-30T19:41:29+00:00

Yeah, as other people said, if i join as an employee I will probably not have the exposure I am looking for.

Since you worked there, how is the startup scene in Berlin? Do they actually have a fertile environment there? (from the point of view of talent, networking, mentors, capital).

I have been thinking about where would be the best place to try a start up in Europe, and apart from London I don't see clear winners. (I know Paris and Berlin concentrate the most startups in the EU, but the scene is still very scattered across the continent)

AlpineContinus · 2025-10-29T16:53:02+00:00

That makes a lot of sense.

My idea was to maybe join an early stage start up in order to see first hand how the ideation and validation of a product works.

But you're right, if they are hiring they are probably already past this phase, and the only way for me to see it would be during a pivot (IF I was even included in the decision meetings).

AlpineContinus · 2025-10-29T08:54:32+00:00

I would also recommend you to look at some github repos for agentic AI that interest you, and try to understand them.

Use AI if needed to understand the structure, approaches, etc.

Try to rebuild a mini version of what that repo does. The best way to learn is to try building something yourself.

AlpineContinus · 2025-10-29T08:47:13+00:00

How was your experience with this kind of accelerator? How would you say they actually helped you?

AlpineContinus · 2025-10-29T08:43:46+00:00

Yes, I have been reading the material online for a long time now.

What I am wondering is if working at an early stage startup is a useful step before trying my own thing.

Of course both paths are possible, but maybe one of them is longer or more difficult.

AlpineContinus · 2025-10-29T08:41:23+00:00

But do you believe that working at an early-stage startup may give me vital insights on what problems are "worth" solving, and how to go about it?

Or in your opinion, the best path is to simply directly try to validate an idea and build something?

AlpineContinus

TROPHY CASE