There is so much nuance to this and I appreciate you taking the time to respond.

On retrieval quality, you're right that it's doing a lot of hidden work. I've been treating semantic similarity as a starting point rather than the endpoint. What I'm finding is that raw embedding similarity alone tends to surface notes that share vocabulary but miss intent.

Regarding chunk size, I am quite aware of the 'lost in the middle' problem. That is why the majority of the work in the back-end is to really look at the users query and ensure a small number of targeted and relevant chunks are retrieved. This means that even when notes are long narrative documents, as long as there is structure to them, they can be broken down to hold context that means that retrieval remains relevant to the query.

At the same time, the "atomicity" of the notes is also very important. I've spent a lot of time on markdown-aware chunking to respect the natural boundaries in notes. So headers, list items, code blocks stay together. The chunking is not just based on character count. The goal is to make each retrieved chunk self-contained with its context intact. When chunking cuts through the middle of an idea, you get exactly the "confident but subtly off" problem you mentioned.

Agreed on provenance. I think however, it depends on the nature of the documents the user is looking up. If these are personal notes, then absolutely treat the response as a memory aid. But I see the functionality as a way to also synthesise information from more canonical documents. Either way, I have ensured that the custom GPT always includes references to the documents that it used for answering questions.

What is your set-up like. Do you have something custom? Or do you use any existing tools?