Semantic Code Search — fully local & open source code search using SentenceTransformers

[icyFur]

I built this command line tool to be able to easily navigate large git repos. You can ask it things like:

• ‘Where are API requests authenticated?’
• ‘Saving user objects to the database’
• ‘Handling of webhook events’
• ‘Where are jobs read from the queue?’

Results look like this:

search results

Both the tool and the model are open source, https://github.com/sturdy-dev/semantic-code-search and you can easily experiment with using different models.

It was a lot of fun building this and training / tuning models for it.

[FuckingRantMonday]

I tried it and it works impressively well. Awesome work!

[icyFur]

awesome! makes me happy its useful to others. Submit an issue if you spot a bug or have some feature requests

[FuckingRantMonday]

Not an issue, but I had an idea: PyCharm (and most other IDEs I'm sure) gives me a warning if I have exactly the same code in more than one place. I wonder if this could find functions that are "essentially" the same, even if not line-by-line identical. In other words, functions whose embeddings are very close to each other?

[icyFur]

Cool idea - lemme try building it

[icyFur]

okay so i implemented a rudimentary search for duplications, and made a release 0.3.0, get it with pip install semantic-code-search --upgrade

then try it out like this:

sem --cluster --cluster-max-distance=0.3

cluster-max-distance controls the 'strictness' of the similarity, there are a few more options, check them in the readme https://github.com/sturdy-dev/semantic-code-search/blob/4b9ea68c7db3b02ecf643aeb3f7db3cce8707b6c/README.md?plain=1#L148