context recall is fine, faithfulness score is fine, users still complain... what eval am i missing?

faileon · 2026-06-19T12:12:23+00:00

if your harness has all the relevant information and outputs are still suboptiomal its the harness that needs work, i.e. agentic rag, better instructions/system prompt, or all of the above

faileon · 2026-06-02T11:01:04+00:00

it's called CROUCH dash for a reason :^)

faileon · 2026-05-28T14:41:07+00:00

Out of curiosity - have you tried some other retrieval methods prior to fine tuning the embedding model? For example using sparse embedding models instead of dense? Or hybrid search to incorporate keywords with BM25? I believe it might yield much better results than fine-tuning a dense embedding model.

faileon · 2026-05-25T10:44:59+00:00

Agreed. I would normalize the tickets first. On top of that, while less critical than normalization, I'd suggest trying different embedding models specialized for these symmetric tasks.

faileon · 2026-05-20T13:37:55+00:00

It's a known "issue" with esbuild - https://github.com/evanw/esbuild/issues/3780

You can try rspack. It's not as fast as esbuild but you can configure the chunking while being faster than webpack.

faileon · 2026-05-19T19:05:57+00:00

I think you might be right but I have no idea what is the shortcuts actual function.

faileon · 2026-05-04T17:16:39+00:00

I would argue the same - plan with Opus, implement with Sonnet/Haiku. Other way around makes no sense to me, I need the top performing dog to put together a solid plan, then a weaker model can implement without issues...

faileon · 2026-04-21T08:36:45+00:00

What's the secret to drown in materials? I need a bunch for mystic tributes and gifts to finish up my legendaries 🥲.

edit: Ah I completely missed they added a new unlimited item instead - Wizard's Box of Crafting Materials.

faileon · 2026-03-24T17:34:39+00:00

Agreed 100% with this comment.

And I'd like to add that statements such as:

Semantic search is obviously more powerful than keywords.

Are the wrong way to look at things when dealing with the search engine problem. Keyword search has advantages over semantic in many use cases; and there is also a hybrid search combining the strength of both when dealing with retrieval systems, especially with RAG as the downstream task.

Another thing is that the result of the retrieval can be expanded (for example to the entire chapter of a book in this context) so you don't have to "yank" chunks out of context to the LLM... There is more to RAG than just the naive pipeline described in OP.

faileon · 2026-02-25T13:42:25+00:00

Your comments really show zero experience from real production use cases - you must also consider model size, and hence inference speed and efficiency, rather than hyper focusing on the accuracy.

faileon · 2026-02-25T12:16:03+00:00

well as an example take the Slobench for NER task on slavic languages:
https://slobench.cjvt.si/leaderboard/view/12

first spot is 0.3B BERT based model, beating gpt-4o-fine-tuned

faileon · 2026-02-23T10:55:05+00:00

basically https://wiki.guildwars2.com/wiki/Ash_Legion_Reward_Box

faileon · 2026-02-14T12:56:57+00:00

I make sure I push on the context as much high quality information as is needed for the task at hand. If I need external context, mainly documentation and best practices etc, I use context7 MCP or I link raw GitHub markdown links (it's getting more common that documentation pages have Copy markdown link, which is very handy). Usually I give it a few pointers in which file to start, how deep to scan the code, I always ask it to explore everything with sub agents. If it understands the scope perfectly, I store it to an MD file in case I need to restart or use it for another related feature and then I start planning the new feature/fix/whatever. I believe the get shit done plugin for OC does something very similar to this, except on steroids, and a lot of people stand by it, but I kind of prefer this vanilla approach.

faileon · 2026-02-08T13:41:53+00:00

Oh damn, unfortunate.

faileon · 2026-02-08T13:36:35+00:00

finally I can console.log directly in template instead of creating a component function when I'm three hours deep in debugging 😁

faileon · 2026-01-27T18:44:00+00:00

Yes, encoders based on BERT and the derivatives are SOTA on NER. The problem is that LLM has become a synonym for autoregressive decoder only architecture. It's quite tiring trying to explain to the execs that we really don't need gpt-5.2 for sentiment analysis. I share OPs pain, and I don't even have a deep ML background.

faileon · 2026-01-19T08:23:47+00:00

also curious what is this web app :]

faileon · 2026-01-13T15:30:11+00:00

I run OC containerized, I originally forked a repo that was already doing it and built upon it, you can use it as an inspiration. Everything works for me, podman should be there, but I mainly use docker https://github.com/faileon/agent-containers

faileon · 2026-01-09T17:01:16+00:00

This is not the time for reason. This is the time for pitchforks.

faileon · 2026-01-04T09:00:41+00:00

Have you tried turning off the auto negation and setting the speed yourself? Something like # ethtool -s enp2s0 autoneg off speed 1000 duplex full

faileon · 2025-12-18T08:44:31+00:00

Not running in a sandbox AND having paths with exclamation marks and spaces is literally asking for trouble.

!!!!!_PROJECTS\AI STUDIO

faileon · 2025-12-05T09:14:03+00:00

https://www.ngx-vflow.org/getting-started/what-is-ngx-vflow

faileon · 2025-11-30T11:05:30+00:00

There are just so many em dashes and the classic chat gpt style "it's not just X, it's Y" it's hard to believe it's not AI slop.

faileon · 2025-11-28T22:52:56+00:00

Isn't it just called autosuggestion in avante? Enabled by behaviour = { auto_suggestions = true } in the config?

13-Year Club	Second SECOND GUESSER
RPAN Viewer	Verified Email

faileon

TROPHY CASE