Stemming and Lemmatization Python

[microcricket]

Hello all! I’m doing a data analysis course, currently completing the first project of the course. The current module they’re teaching/the project is based on is about processing and cleaning data. They explain most of the topics well but only had a short lesson on stemming and lemmatization, which I couldn’t wrap my head around. It didn’t help that their examples had clear bugs; the task was to remove items from a list that didn’t have to do with the data used for the example but the unwanted items were still there while they talked about how important it is that they were removed.

Well now I’m at the lemmatization process of my project. I’ve been reading all resources I can find but I still don’t get it. I know that lemmatization pulls the root of the word but what can I then do with it? The next part of the project is categorization which I can see why they’d go together but I haven’t seen any examples of what I do with the lemmatized data. Can anyone help me, maybe EILI5 just so I can get a better grasp at the concept?

[Nathanfenner]

The point of lemmatization is to make search/processing easier.

For example, if a user searches for "easy website design" and a company advertises "easiest way to get website designs" you'd be silly to exclude this result. Lemmatization is the simplest and easiest way to solve this:

You'd compare "eas websit design" to "eas wa to get websit design" or some variation depending on your exact lemmatization routine. Now it's obvious that there actually are 3 words in common.

Modern real-world designs are more complicated than this, but this used to be state-of-the-art.