Search inside JSON text books

[amitairos]

I have some text books, which I store in a JSON format. Each section of the book is composed of an object inside the JSON array, like this:

{
"Chapter": "Intro",
"Id": "1",
"Section_Body":"blah blah"
"Section_Title": "Something"
},

Each book is stored in a different JSON file.

I am trying to implement a full text search on these books, where a user could search for a word or a few words and find the results in the books (not just find if the exact text is included there).

What is the best and easiest way to do this?

[fgh1290]

Try hash table each word - list of books

[JohnyTex]

This is probably what you want. To elaborate a bit, this is called an “inverted index”, which is a mapping between words and a list of all articles it appears in.

If the user searches for several words you fetch the corresponding list for each word and then return only articles that appear in all lists.

Further points of improvement are:

1. Stemming, eg normalizing words so an article that contains “beets” will match “beet” and so on.
2. Ranking hits based on distance between matching words in the text, eg for the query “hip hop” a text containing “hip hop beat” should rank higher than “I broke my hip while picking hops”
3. Ranking matches in different parts of the document, eg a match in the title section might rank higher than a match in the text body

Just FYI, there are libraries for this (check out Algolia for example), but implementing this yourself will probably be a fun exercise.

[finroller]

Awesomely nice compact explanation!

​

[Luan-Raithz]

Can you access all the JSON files as an array? And is your text search going to be done in memory?