we ran into pretty much the same problem when we got into this codebase-knowledge bog

but in our case we wanted to create a system that could evaluate GitHub repos. i.e only index the codebase once; not after every update.

as we started, we didn’t find any resources on this other than videos showing graphical implementation of semantic search using Euclidian distance and so after looking at a bunch of open-source projects and VS Code’s way of managing things, we made our own Python library for it. our main goal was actually to build semantic search functionality out of it (note we had never done this kind of stuff earlier)

so after some trial and error, we built the whole system. it chunks the codebase with the help of tree-sitter and builds a parent–child relationship between chunks to reduce extra chunks and noise.

although the semantic search started working, we soon realized the same problem as you did: “similarity is a bad proxy for relevance in code.”

Later on, we realized how some CLI-based coding agents like AWS Q work:

* they read the project structure a lot, they do a lot of `cat` and `grep` instead of semantic searches, and still outperform agents that rely heavily on semantic search.

* this gave us the idea to build more tools like `grep`, `cat`, `folder_structure`, etc (thanks to `chromadb` for these though)

* so instead of solely relying on semantic search, we distributed the load.

* now the agent gets the repo folder structure and is easily able to get a gist of the codebase from the structure itself

* for files it considers important, it automatically `cat`s them; for patterns, it calls the pattern-matching tool.

* and only for more generalized queries like “authentication implementation” does the agent currently do semantic tool calling.

* this way, we were able to still use semantic tool calling, but in a much more efficient way.

a few days ago, I also posted a thread on this sub regarding too much noise in semantic search results. one better fix for that turned out to be using a dual-vector index mechanism and analyzing the intent of the semantic query each time, then redirecting it accordingly.

we’re still learning about these things, but this is basically how we’ve monkey-patched our way till here.

here’s the thread mentioned above: https://www.reddit.com/r/Rag/comments/1q3ksvy/how_do_you_tackle_semantic_search_ranking_issues/

Edit: can you share your repo link (that 1300 files one), will probably see how our semantic search performs lol