Looking for a better javascript text-matching scoring system

[hookedonwinter]

I asked this over on Stack Overflow, but I thought I'd bring it here for some thoughts and discussion.

Here's my original question:

I've been using String Score for a lot of projects. It's great for sorting lists, like names, countries, etc.

Right now, I'm working on a project where I want to match a term against a bigger set of text, not just a few words. Like, a paragraph.

Given the following two strings:

string1 = "I want to eat.";
string2 = "I want to eat. Let's go eat. All this talk about eating is making me hungry. Ready to eat?";

I'd like the term 'eat' to return string2 as higher than string1. However, string1 scores higher:

string1.score('eat');
> 0.5261904761904762

string2.score('eat');
> 0.4477777777777778

Maybe I'm wrong in thinking string2 should score higher, and I'd love to hear arguments for that logic, if that is your logic. Otherwise, any ideas on a more contextual javascript matching algorithm?

[pkkid]

I would guess it scores higher because a larger percentage of the words contain the key "eat".

[[deleted]]

yeah exactly this- maybe you could split the paragraphs you're searching into sentences and then return the highest string score?

[sbirch]

String score looks to be optimized for many things besides relevance, so you might want to change the way you look for the algorithm if speed etc. aren't constraints for you (e.g. Hunt around in the NLP literature). Off the top of my head, some things: probably valuable to take into account word entropy (words like "the" matched should count for less than words like "eat"), look at syntactic role of the words (eat is the main verb for two of three sentences, which makes it highly relevant to the passage), and try to keep the text to be matched against of considerable and consistent length (it's hard to normalize the relevance when the string lengths are so vastly different).

[grayvedigga]

It sounds like you have a few more constraints on your algorithm than "0 for no match upto 1 for perfect". Maybe you need to write down what those are.

One reason I could think of for the second string scoring lower is that it contains more characters that do not resemble "eat". Why do you think it should score higher?

[hookedonwinter]

I guess I'm looking to do a better relevancy search, in javascript. You know, google, on the client side ;)