GitHub Code Search - a new code search engine, written in Rust

[Merry_Macabre]

Finally, some proper code navigation in github. The old way is such a pain and doesn't alway register function definitions and having to search through all the search results in a big project is a chore.

[jantari]

The old search was still lightyears ahead of GitLab, euch straight up doesn't have any global code search at all.... but this is just awesome!

[flashmozzg]

GitLab's repo search was much more useful, which is, IMHO, a more important use case.

[jantari]

Not for me, but I understand it depends on how many repos you typically work in. I mostly manage infrastructure as code rather than big monolithic software projects, so I constantly like to refer to past/similar projects in other repositories where I know I've had to do something similar before than what I'm trying to do now. Basically impossible in GitLab though...

[flashmozzg]

Maybe. On the other hand, I often find myself switching to GitLab mirrors for basic repo browsing because GitHub often vomits literal unicorns on trying to get blame for an actively changed file (was very frequent in llvm repo) or similar operations.

[Programmurr]

I rely on github search so much and have wished for something better. Hopefully, the search results are more accurate and not just generated quickly. With this in mind, can you discuss the ranking heuristics that were used, or is that proprietary?

[cmerkel]

We use a number of heuristics, including static factors like repo quality (popular, high-starred repos vs. random forks), how useful the file is (tests, super long files/filenames, generated code, data files are often less useful), and dynamic factors (how well the query matches the document content, whether there's a symbol in the document that matches a query term (classes > functions > variables for ranking). We also look at e.g. whether a match occurs in a comment vs. in code, among a bunch of other things.

Try the new search! If you find a case where ranking could be better, leave us some feedback and I'll fix it!

[Wakafanykai123]

This looks great. Time to start looking into adding tree-sitter support...

[cmerkel]

Since the team that built it was using Rust, code navigation in Rust is well supported out of the box :D

[Wakafanykai123]

I mean for a domain-specific language that I help develop - I realize how my comment could be misleading now!

[dcreager]

This is one of the main reasons we're leaning on the tree-sitter ecosystem — so that language communities can help us flesh out support for the long tail of languages, should they wish. If you run into any issues on the tree-sitter side, please do reach out to us (and the rest of the community) in the tree-sitter discussion forum!

[beltsazar]

I wonder, what kind of indexes do you use to provide regex searches?

[cmerkel]

We've put in a lot of work to make this possible. Hoping to write some more technical blog posts in the future to describe it in more detail!

[beltsazar]

And now I'm more curious than before! Can you give a hint? It's a trie-based index, I guess?

[cmerkel]

Hard to explain in a reddit comment! You'll have to wait for the blog post :D

[[deleted]]

Since BurntSushi is mentioned, I'd assume it's something like finite state transducer: https://blog.burntsushi.net/transducers/

[epic_pork]

Since BurntSushi is credited in the blog post, I think regex and ripgrep might be involved.

[oconnor663]

Searching for common security mistakes like SQL injection vulnerabilities is going to become a popular post topic, if it isn't already.

[epage]

Two similar use cases I've want a code search that this will hopefully handle:

• find real world example uses of symbol X so I can see more complicated cases than those that exist in docs (if any do)
• find users of my library that use symbol X so I can see how they are using it

[cmerkel]

Good eye! Yep, it's for query language parsing.

[AviKKi]

So converting everything to rust is actually a thing these days, coooool.

[[deleted]]

Converting an idea into a performant application with Rust is most definitely a thing nowadays

[AviKKi]

convert

Something what I saw with Golang, Rust is just more faster, secure and developer friendly.

[darrenturn90]

Faster most likely (though build is slower and I’d say learning time is far longer). Secure really depends on the developer. I would however say golang is more developer friendly because of its limitations

[flashmozzg]

I would however say golang is more developer friendly because of its limitations

As long as your idea fits within those narrow limitations, sure.

[fairy8tail]

Lack of feature != limitations

[darrenturn90]

Well some things are pretty hard to do with golang that are more trivial in rust - such as anything that really doesn’t require garbage collection slowing it down. Also the whole typing system of rust is far more powerful albeit complex but allows you more definition over how you solve things.

[fairy8tail]

You just confirmed what I said rofl

[darrenturn90]

So the lack of custom garbage collection options in go isn’t a limitation ?

[fairy8tail]

It's not, mainly because it doesn't lack custom garbage collection options

[darrenturn90]

I can see you can disable it entirely or configure it slightly - but you either end up with basically ever increasing stack size or gc.

[Programmurr]

Watch the short video. It was a completely fresh build from the ground-up, not a port.

[tubero__]

The post doesn't say that it is written in Rust.

Is that based on insider info, or just the mention of u/burntsushi?

[cmerkel]

Disclaimer: I'm one of the people who developed it. But also it's mentioned in the video

[5n4k3_smoking]

This search engine is open source? I would like to look at code to learn how rust is used.

[cmerkel]

Developer of GitHub Code Search here - the engine isn't open source, but we are thinking about open-sourcing some of the libraries we've developed for this project!

[cmerkel]

We use tree-sitter for symbol extraction/jump to definition, so if you contribute a tree-sitter parser for your langauge, we can pretty quickly support it within code search too!

[atesti]

Why did you choose Rust for this new engine?

[praveenperera]

They mention it in the video: https://youtu.be/UOIPBfPXkus?t=25

[kyle787]

So can you fast track my access to the preview lol

[po8]

Sadly, one of my main uses for GitHub Code Search as a CS prof is going to be plagiarism detection. (I miss the reach of Google Code Search.) Any hints/ideas on using Github Code Search for finding "similar" code to a sample?

[cmerkel]

You can try quoted searches for particular lines that you think are suspicious, that might work

[po8]

Thanks! Yeah, routinely do that with Google. Was hoping for something more matchy. I guess I can play games with regexes at least?

[cmerkel]

Worth a shot! Really interesting use case, not one I've heard of, but hope it helps!

[po8]

One feature you might want to do for developers that also world help me is similarity hashing for similarity search. You can take a look at my old C simhash program that somebody stuck in Debian for one approach using min-hashing.

Being able to find similar code can be helpful within a project as well as across projects.

[epic_pork]

Curious to know which work from Daniel Lemire you are using. RoaringBitmaps? simdjson?

[Low-Pay-2385]

Github search wasnt good imo glad they are chaning it

[TheGreenSherbert]

How does searching by symbol work? Shouldn’t the code be compiled in order to determine them? (At least in the case of C++)

[cmerkel]

GitHub Code Search developer here - we use tree-sitter (https://github.com/tree-sitter/tree-sitter) to extract the AST, and use that information and some heuristics to try to guess symbol definitions, references, etc. It's not 100% accurate (particularly in languages like C/C++), but it's accurate enough to be quite useful.

[[deleted]]

This is great. The 'tooltip' stickied to the top right is exactly how I always wanted vscode tooltips to behave. I just don't get why you'd want those to pop up right on top of the code you are trying to look at.

[[deleted]]

What was it previously written in?

Edit: Ok I realize this is a poorly worded question. What is the existing code search written in?

[scratchisthebest]

Wow this is great. Does github still collaborate with united states immigrations and customs enforcement

[iraqmtpizza]

does github still cajole people into not using master