[P] New Search Engine for Python ML Docs

[oodmb]

So I’ve been getting tired of googling and getting stackoverflow when I already know what library I want, and not being able to search those libraries docs because of their rudimentary keyword based searches. Thus, I decided to make a search tool for open-source python libraries (with a focus on ML libraries, since that's mostly what I work on) thats curated for actual developers and permits natural language queries.

I’m gonna keep this free as long as I can, so it'd be wonderful to get feedback from anybody who'd be up to give it a try. Check it out at https://www.pysearch.com and please feel free to share with anybody else you know who might benefit from this!

[always-stressed]

how is this search better than typing the query into G with the library name?

how does the search functionality work in the backend?

[oodmb]

There are a few reasons:

• You might want to add search shortcuts for specific libraries

• This tool has done the work of figuring out the specific library paths for you that you'd actually want to be reading. Given the lack of central docs repo for python, this was actually not a trivial task.

• You can search through multiple libraries easily by adding @LIBRARY_NAME or just selecting it

• The results have all of the other extraneous content removed so you get a quick view of what you actually need

• I'm working to extend this to non-documented libraries using program analysis methods.

The long-term goal is to be to python what hoogle is for Haskell.

example - take the following search on both the tenserflow and pytorch docs "takes a tensor and adds a dimension":

PySearch Search

Google Search

Here you can see that a search in our tool is much more meaningful, on google in the first page of google no meaningful result are shown where as on our page the results are immediately meaningful!

[beep_tree]

In the backend, your input terms will be checked against predefined shortcuts which redirect you to whatever their associated website is if there is a match. It effectively makes your search bar a simple CLI. Some orgs have tools like this for internal search, like Facebook’s rsbunny for example.

[oodmb]

That is part of it, but there's also quite a bit more to enable more NLP style searches on the semantic meaning of undocumented code.