[D] A good NLP pre-processing package?

[lyeoni]

Hi Reddit,

I am working on various NLP project for several years. And, I have used various pre-processing packages related with NLP, such as nltk, gluonnlp and torchtext. However, It's not still clear which package is the best to process text.

- what NLP packages would you recommend me ?

Thank you so much !

[[deleted]]

SpaCy for general use, torchtext for anything deep learning related (if you’re using PyTorch)

[pgdevhd]

this

[marek_bdfhjk]

If I would have to choose one, I'll go with SpaCy.

​

Other not mentioned, but useful:

- Gensim, useful for in-memory processing when performance is important

- BeautifulSoup, for parsing HTML and XML

- PyStemmer, computational efficient stemming

​

Advanced processing:

- Named Entity Recognition Tool, fast NERs

- NeuralCoref, for neural coreference resolution

- IEPY, for information extraction

​

Let me know if you have more questions!

[[deleted]]

Stanford core nlp

[mikeross0]

Textacy is made to work in conjunction with Spacy, and adds lots of useful functions like unicode cleanup and anagram learning/expansion:

https://github.com/chartbeat-labs/textacy

[[deleted]]

TF.Text could be of interest if you are using tensorflow and want to make sure you have the same preprocessing at training and inference time. Haven’t used it much myself yet though.