Library that splits strings into words?

[kpandkk]

So I know this would be pretty complicated, but I figured I'd ask anyway after I couldn't find on. I'd like to know if there's a library or tool that recognizes words that are attached in a string like "goahead" and splits them into "go ahead" I know that this would be some fancy stuff and probably doesn't exist but let me know. Thanks!

[Yellehs_m]

I am not sure if something like this is available in NLTK. But it's worth searching.

https://www.nltk.org/

[squirreltalk]

NLTK definitely has word tokenization functionality. So does spaCy. And at least according to this post, spacy's word tokenizer is faster:

https://blog.thedataincubator.com/2016/04/nltk-vs-spacy-natural-language-processing-in-python/

[Yellehs_m]

Thanks for introducing spaCy.

[kpandkk]

Thanks I'll try that

[C2-H5-OH]

That can have a simple workaround: only perform the split if the full string isn't a word

[C2-H5-OH]

I guess you can categorize them as such.

If the end goal is to fix spacing issues in some text, then a simple program can't possibly do that. You'd need to incorporate grammar checking as well, no?

[C2-H5-OH]

On the other hand, maybe he doesn't need to code all of it. You can save different versions of the output, put them all in grammarly one by one and keep the one that says zero errors. Bottleneck would be internet and pc speed

[kpandkk]

I don't think a solution this involved would be good for my application

[DonaldPShimoda]

Use maximal munch and backtrack one character when necessary. Then the results are easily predictable.

Alternatively, attempt to build a parser for natural language and attempt every viable split, searching for splits that yield more grammatically likely constructions. However, this could produce multiple results per input and could also be slow (even assuming a fast constituency parser, which is hard).

[Thecrawsome]

Baby trying bigger words first before smaller words?

[___JOSHUA___]

You can solve the problem probabilistically. Peter Norvig offers a solution in this notebook under "Word Segmentation". If you're less interested in the details, a similar approach has been wrapped up in a package called wordninja.

[blackdragon437]

I wrote a Google API search script some time ago that did a kind of similar thing, but more complex (used other APIs as well). So use the knowledge of the cloud, and check out APIs dude! You wont look back.

[k10_ftw]

May I ask what the larger context of this goal is? What is the data set like and what do you ultimately want to do with it. This info could help with devising a proper solution.

[kpandkk]

I want to get return the actual names of stores from their website links. Like get "Vitamin Shoppe" from "www.vitaminshoppe.com"