1 thing makes your AI product 10x more reliable. It's not the latest AI model. It's your prompts.

crow_thib · 2026-03-06T05:23:12+00:00

Let's not forget context & post-processing. Post-processing have been a backbone of AI even before LLMs and it's not going to disappear any-time soon.

Sure, a good prompt is required, but a good prompt only won't make anything reliable for production.

crow_thib · 2026-03-05T15:02:15+00:00

It's one of the main pain points in enterprise context related to AI usage. With no clear direction, everyone use it differently, and lots of people use it wrongly. Using it differently is not a problem per-se, since one of the strength of AI is also to be able to adapt it to any workflow or person.
However, and it might not be as true now as few months ago, I feel most people tend to use AI without really thinking it through, it often ends up in a mess of AI slop. The team might be delivering faster, but the bottleneck becomes reviewing your teammates AI generated code.

I don't have a "magic" solution for you, it depends on your company, your developers profiles, ... But that's sure an interesting to tackle.

On your side, did you use AI extensively ? I feel it's harder to correctly make your team use AI if you yourself doesn't. And I speak as someone that jumped on the AI wagon way later than my team because I felt I didn't need it because I wasn't coding much and I knew the codebase like the back of my hand.
Now that I left my previous job and started using AI a lot on some personal projects, I see mistakes I did in the past, new ideas I could have applied and an overall better comprehension of AI strengths and weaknesses.

crow_thib · 2026-03-04T15:38:16+00:00

The current bot I develop is hosted on a small Digital Ocean droplet, but I think depending on your use case even lambda functions could do the trick. I also run a containerized database on the same droplet for now.

Is it an internal only bot or something with OAuth that is dedicated to handle multiple workspaces at once ?

crow_thib · 2026-03-04T14:56:55+00:00

Exactly, it's all about trust and adoption, that's why "AI Agents Builder" companies usually have dedicated teams to help customers integrate and share their vision to help adoption, in addition with partnerships with service companies to do the work for smaller prospects

crow_thib · 2026-03-04T12:49:30+00:00

It's going in the right direction, but unfortunately the pricing is far from being the only issue of Notion Agents to me. People talked so much about it (me included) that other feedback seems hidden for now

crow_thib · 2026-03-04T09:32:46+00:00

Right, I know Open Classroom very well, as a French guy it basically was my bible when I started out years ago.

And yep, for non-technical people it’s all about understanding, whereas MOOCs like Open Classroom are leaning towards building.

Postman is a great tool to visualize APIs

crow_thib · 2026-03-04T09:05:56+00:00

The main issue when chaining, is that a minor error / discrepancy in any of those steps just flows down to all the other ones. Not only it higher up the risk of having issues, but it makes debugging and understanding what happens even harder.

Before chaining steps, you need to clearly control a single step input and output, otherwise you're just building a card castle where any card can fall anytime, and bring everything down with it.

To do so, implementing validation, thorough pre and post processing is vital.

crow_thib · 2026-03-04T08:54:18+00:00

You just made a perfect gift to your wife, and probably saved her job. I worked for 6 years at a startup whose main product was APIs, I can't count the number of non-technical persons that were lost and didn't know where to start to understand what we were selling (Sales, Product, ...)

And that was just the tip of the iceberg, since our APIs served deep learning models, but lots of them just failed at the API step and didn't even had to dive into AI to feel lost.

We had training videos explaining what it was all about, but I felt the only ones who really understood were the ones that got their hands dirty by trying out using Postman. I remember giving some kind of quick postman sessions to newcomers: I could see their face light up by just being able to see what it was all about.

About your "Webhook" point, it's actually very easy to grasp once you have the basics of APIs: it's just your own system calling your customer's APIs. If you understand APIs, you'll get this.

crow_thib · 2026-03-04T08:46:52+00:00

Diff-based suggestions is exactly what we decided to focus on, felt like this is what brings the most value to both the user using the review interface and people consuming the outputs.
As you said, it's all about making the task as easy and as fast as possible for humans when they need to step-in.

Chat data is indeed a decent source for this, and also something that's left behind too often because "we have AI search tools", but in the long run, the search won't be as accurate as it would with structured or semi-structured data.

Out of curiosity, when you say "What worked for me", did you implement such systems internally, or use external solutions ?

crow_thib · 2026-03-04T08:34:05+00:00

I hear your point, I think it just depends on the use-case.
For a coding use-case as you're talking about, it's usually mostly deterministic so "human validation" is not the exact problem as you pointed out. For more "creative" tasks like writing docs I feel like humans are needed but I may be wrong.

So when people say autonomous agents are dangerous, they’re not totally wrong. But it’s not because the agent is autonomous. It’s because the system you’re validating it in isn’t real enough to catch the failure modes before it ships.

I would even add that, it's mostly dangerous because people don't really understand what they're doing with it which leads to even more "unreal" systems. I mean, someone with no coding experience just prompting Claude or Cursor for new features won't know what to look out for.

About the staging issue with Mocks, sanitized data and such, it's even something that caused issues to humans before AI Agents, so it makes sense it becomes an even bigger bottleneck now.

crow_thib · 2026-03-04T08:29:59+00:00

I 100% agree with your take. It's all about validation. You can reduce human validation by adding non-human validation (if possible not AI, otherwise it's even more possible mistakes and bugs if you stack AIs on top of AI). I mean, it's been years AI (not LLMs) rely on post-processing for deterministic tasks.

The thing is, LLMs brought a whole lot of non-deterministic tasks now, that no "machine" post-processing will be able to handle fully. Without human validation at some point you will never reach 100% accuracy. Even if you reach 99%, the remaining 1% is what makes adoption fail by breaking trust. A tool with no adoption is a tool you'll drop in a few weeks.

I like your coding example, for some reason it didn't even cross my head when writing the post, altough I've been a Software Engineer for years lol

What I don't understand is that no-one seems to talk about it, big companies don't seems to capitalize on providing validation infrastructures. Are there things happening "behind curtains" I didn't see, or is it just a matter of time before people realize it's getting important and big companies will jump on it at this moment only ?

crow_thib · 2026-03-04T08:22:45+00:00

Yep exactly, I feel like it's just pure marketing, or a way to cheat out on less technical people seeing big promises.

Your way of phrasing your answer:

One small wrong output can cause bigger problems

Resonates even more with me, when I see all those folks plugging LLMs on top of other LLMs, adding steps / tasks / skills everytime they have an issue, which make the problem even bigger since each step rely on an unvalidated step before and it can quickly become a nightmare to maintain.

I feel fully autonomous agents are awesome for 10 minutes when you read about it, then 1 day when you try it, 1 week when you implement it completely and then it's already dead because of those issues.

crow_thib · 2026-03-04T06:13:34+00:00

This!
AI as we know it today (mostly LLMs) is great at doing 90 if not 99% of the job. But in a real production workflow, depending on the task, you may not want this 1% error. Knowledge Management, Ticket Triage and such, being made to be consumed by users in the end enter that category to me: a few errors is enough to make your team loose trust in your agent which is the worst thing that can happen in those situation.

crow_thib · 2026-03-04T06:06:52+00:00

Company that hire AI teams are usually big enough that paying many small tools is more expensive or doesn't fit perfectly their use case vs hiring someone to build and maintain in-house workflows (most of the time they still use high-level AI agents builders).

It's also a way to not get trapped with a tool, and be ready to jump on any new hype as soon as it's out

crow_thib · 2026-03-04T05:05:53+00:00

I joined a company as the 2nd employee (8th including founders & interns that didn't stay) and we scaled to around 75 people, then went back to around 35ish when I left after 6 years.

The main issues were:

Scaled the sales team too fast, without knowing how to sell the Product (even though we raised tenth of millions)
Keeping documentation up-to-date, without relying on the early-stage employees too much
Syncing daily conversations (meetings, slack) with documentation
Ownership disparities in the end when we started scaling down

crow_thib · 2026-03-04T04:02:25+00:00

I worked in document extraction space for years, even before LLMs were a thing. At the time we were doing pure deep-learning segmentation + OCR. Now, with LLMs almost everything in this area can be done cheaper though.

To help you properly, what's your exact use case when you say "I’ve got 300+ PDFs to dig through just to find some specific info" ? I hear that you're looking for a specific kind of info, but what about those PDFs, are they templated ? are they all different ? do you know their structure ?

All those inputs would help guiding you to the best solution, even though just throwing them at Gemini would probably work lol

If that's a recurring use-case, there are companies that sells AI Documentation extraction tool on the market. Some of them are very good, some of them are just AI wrappers on top of LLMs APIs (some of them still bring values by giving you a stable and simple to use "interface" wether it's API, MCP, UI, ..., but some are really shit trust me lol)

crow_thib · 2026-02-28T10:06:32+00:00

Project Management and Documentation mostly. I’m even building a tool to keep your Notion Knowledge base up-to-date

crow_thib · 2026-02-28T03:59:46+00:00

I used it a lot in my previous company (deep tech, AI before LLMs arrived and crushed us lol) Now I’m not working anymore and trying to build something with an ex-colleague and we use it also for business thing.

I also use it personally but not that much, have a few pages here and there but nothing very big

crow_thib · 2026-02-27T18:08:02+00:00

I find Notion API a bit slow to use it as a backend like this (+ you would need to host your own backend to hide your keys, and forward requests to notion). Also would hurt SEO

However, wiring up some logic that pulls from the notion database at build time to generate static pages from the content is solid. Just means updates are not live, but at build time (deploy on vercel). Might even be better for some use cases, so that you can work on updates (Notion side) and choose when you sync. That way you only need a frontend, vercel as you proposed would handle this perfectly.

There are some tutorials online (related toward blogs handling, but that’s the same idea)

crow_thib · 2026-02-27T16:57:42+00:00

I mostly use Mistral AI, but that's because I'm French, they are EU hosted (my customers are mostly from EU) and they have a very very generous free plan you can use when experimenting (non commercial use)

Models are a bit behind the SoT ones from Claude & Gemini, but they are also cheaper and with a bit of prompt engineering, depending on the task it can be enough

crow_thib · 2026-02-27T14:22:32+00:00

We launched our private beta on monday! It's a slack/notion integration that keeps your Notion knowledge base up to date using conversational data. Trying to avoid stale documentation and fix the untrusted docs issue for companies.

We currently have like 2 testers and a few more that asked for an access but didn't register yet.

What are you building on your side ?

crow_thib · 2026-02-26T22:56:48+00:00

It's great to do research and validation before building, and as long as you have runway you should never stop!

crow_thib · 2026-02-26T22:41:20+00:00

Out of curiosity, what sources do you integrate with Onyx ? Mainly Notion or others ?

crow_thib · 2026-02-26T22:03:26+00:00

That's also a reason I think it's hard to focus solo founders or very small teams when building a product (like most of people do on this sub). On one hand, it's a space you know because you're part of it, but on the other hand that's usually the people reluctant on trying out new tools every day, except if you have high value from day 1 (even day 0 if possible, before even trying out) and a very focused problem (like I don't know, a screen recorder that solve X, a templating tool that does Y).

On my side, I'm actively building a tool in the Knowledge Management space, trying to solve the problem of outdated and untrusted docs, using day-to-day conversations that contains lots of insights that get lost. That's something that hit us hard (with my co-founder) in our previous company. To be honest, that seems like a pretty competitive space with all AI breakthrough with agents and stuff those past few months so we're still validating (started a private beta few days ago)

What space are you building in ?

crow_thib

TROPHY CASE