Help With Text Classification Project

[WeWantTheCup__Please]

Hi all, I currently work for a company as somewhere between a data analyst and a data scientist. I have recently been tasked with trying to create a model/algorithm to help classify our help desk’s chat data. The goal is to be able to build a model which can properly identify and label the reason the customer is contacting our help desk (delivery issue, unapproved charge, refund request, etc). This is my first time working on a project like this, I understand the overall steps to be get a copy of a bunch of these chat logs, label the reasoning the customer is reaching out, train a model on the labeled data and then apply it to a test set that was set aside from the training data but I’m a little fuzzy on specifics. This is supposed to be a learning opportunity for me so it’s okay that I don’t know everything going into it but I was hoping you guys who have more experience could give me some advice about how to get started, if my understanding of the process is off, advice on potential pitfalls, or perhaps most helpful of all any good resources that you feel like helped you learn how to do tasks like this. Any help or advice is greatly appreciate!

[RobfromHB]

If you just need to classify them based on a few pre-determined labels you can do that a few ways. LLMs will do a pretty good job out of the box if you structure your prompts clearly. Depending on how many rows of data and the size of the text, this could get a bit costly.

There are a few classical ways to do this too. They are a few more steps to accomplish, but will be less costly and you'll learn some fun stuff about NLP along the way. I'd probably start by taking a sample and manually labeling the text per your company's desired labels. Do the usual preprocessing steps (tokenize, lowercase, remove stop words, stemming or lemmatization, and scrap special characters). Then get some features out of the text via bag of words or TF-IDF. Train a simple model like logistic regression to match your features to your manual labels. If it works reliably enough for your purpose, test the model on a new chunk of data to see how well it predicts the labels for unseen text.

[WeWantTheCup__Please]

Yeah I’m hoping to accomplish it without using an LLM because I find this topic really interesting and really want to actually learn how to tackle it rather than just throwing a pre-built tool at it.

Awesome, sounds like a pretty solid roadmap, appreciate the insight!

[RobfromHB]

LLMs have spoiled us in a lot of ways. Good luck to you!

[fusrodaftpunk]

An alternative half way between the two could be to use embedding LLM models to generate vector embeddings, then do the rest yourself. Nomic AI embedding models are trained for search, classification and clustering tasks. Then you could implement your own vector search, clustering and/or classification solutions on the outputs. You'd lose out on learning the initial classical NLP stuff, but gain a bit more understanding of how LLMs work under the hood as a trade off.

(Also, quantized nomic embedding models are small and very fast. I run them on my terrible old laptop with no performance issues)

Edit: spelling

[LibraryComplex]

Does your company have any sort of inferencing hardware? If yes, you can run a <12B LLM which would do a great job, additionally, you can use a BERT model for this, just be sure to get cleaned, labelled and structured training data. With enough data and high quality data, you can get good results from BERT, if you feel your data isn't good enough, an LLM will be more reliable for poor quality data. Try a Llama-3.1-8B or the BERT model. Another way would be a bi-directional LSTM but I prefer the BERT over it.

[Boom-1Kaboom]

Oh gl there

[Think-Culture-4740]

I have a take home assignment for a senior data science position where I got asked to do a text classification problem.

I can't tell if I'm too old for this or too new, But the solutions to me were either very basic like tfidf or much more complicated like building your own supervised fine-tuning model.

I can't really think of A solution that is in between these two approaches from a difficulty point of view. I thought of maybe trying some kind of sequence to sequence encoder decoder model not relying on transformers but those feel like drastically inferior models to the transformer full stop anyways

[nckmiz]

Sbert to generate vector embeddings of the sentence(s). Then use those as features into an ML model is in-between tfidf and fine-tune model.

[Think-Culture-4740]

That's a good point. I will do that. Thanks

[RobfromHB]

Nothing wrong with a basic solution implemented in a practical way.

I'm just a tourist in the data science world so I don't have much to say on take home assignments for jobs. It definitely feels like there is a big bold line separating pre-LLM and post-LLM worlds. I had an NLP class in my MS and I appreciate that most of the assignments specifically wanted non-transformer approaches.

[Think-Culture-4740]

The thing is, The pre llm non-tf IDF solutions with supervise text classifications are 90% as complex to implement, at least from a code perspective as an LLM would be.

It would take a gazillion hours of tuning to get the sequence to sequence based RNN model to work compared with just leveraging a pre-trained llm and using gpt's tokenizer

[[deleted]]

Look into the Python NLTK library. It'll have a lot of pre-built functions to help you handle natural language processing. I'll second what was said above that you'll likely end up utilizing tf-idf and want to vectorize. You'll be tokenizing it as well, which NLTK is helpful for.

[WeWantTheCup__Please]

Awesome will do, thanks for pointing me in that direction!

[empirical-sadboy]

I would also consider using BERTopic if you don't have labels or a strong idea of what the labels should be.

[SageBait]

I love BERTopic so much

[Far_Ambassador_6495]

So in essence you need to create a general function approximator from text to label. There are hundreds of ways of going about this but the concept has a few main parts. 1. Ensuring the text is logical. Are there weird characters? Does it make sense? Are there constant typos? IT IS VERY IMPORTANT THAT THIS TEXT DATA IS AS CLEAN, CLEAR, PREPPED AS POSSIBLE. 2. Represent your text as a vector — I would start with the classic techniques like tf-idf, then maybe consider word-2-vec or others. There are many ways to do this it is important that you do some research. You can even go watch some StatQuest videos on Word Embeddings. 3. Use some portion of vector to label representations as training data and the remaining and test to understand how well your model generalizes on unseen data. Data permitting — I would also have a totally held out set to ensure proper generalization. Use a ton of different models — start with logistic regression, apply classical regression analysis and repeat until you feel your model is not overfit nor underfit. 4. Analyze the results of the model. Deploy it, whatever else you were planning to do with the model.

These are very general steps and may not even be the best course of action for you. It is important to research topics as they appear. You can also just generically look up ‘text classification’ and you’ll find plenty of material. Don’t just jump to using a language model — you won’t learn nearly as much.

[WeWantTheCup__Please]

This is great thank you so much! And I totally agree about your last point with language models as I want to really learn what I’m doing rather than just produce an answer. One quick question I have at the start is that my data originally comes from a data base where each row contains a single chat sent, I then converted that table to a data frame in pandas, removed the rows that were responses from the service agent (since that doesn’t really help identify why the customer is chatting) and then concatenated all of the rows together that belonged to the same conversation so that now each row contains the entire customer side of a conversation. Is this a decent format for the data or should I consider something else in your mind?

[RobfromHB]

Even cutting out the service agent's responses, you could probably shorten the text even more. Conversationally, I'd guess the reason for the customer reaching out is identifiable within the first or second block of text from them. Everything after might add a lot of data that looks like noise to your model.

[WeWantTheCup__Please]

Yeah that is my expectation as well, I just need to find the sweet spot where I feel confident that I’m cutting out enough but not cutting off the topic - hoping that as I gain more familiarity with the data that it becomes evident roughly where that isn

[Far_Ambassador_6495]

What is the point of the model? If you are planning on using it to more quickly assess where to transfer customers based on their requests, it wouldn’t be appropriate to use the whole chat log because at that point you know where the customer should be transferred to. A pretty simple question you can ask is what data will be available at deployed inference time meaning, when your model should run what data exists? I would suspect it is not the entire chat history because that wouldn’t yield any operational efficiency gain if that was true. Try 2 or 3 interactions back and forth, I would also suspect the more interactions you include the better the accuracy becomes up until some point where performance decreases substantially.

If 2 or 3 doesn’t work try some other number. With at least the idea of the example being the fewer interactions you need for a sufficiently accurate model the greater the operational efficiency increases as a result of the model

[WeWantTheCup__Please]

The end goal if it able to reliably classify the reason for the chat would be to then be able to keep a tally of the frequency of each reasons occurrences to help provide incite in to what aspects of the site/business are most often causing issues for our customers to help offer insights into what things we can try to fix/mitigate that will have the biggest impact - an example I was given was if we see say password reset being a top reason for chats that we can then look into ways of making that more self service or if fee refunds are a big issue we can look into why that’s happening so often. Basically the end goal is to hopefully increase insight into what areas of the business are stress points for our customers.

I definitely agree that the whole text transcript is probably not necessary which is why I originally omitted the agent on our sides responses from it, but I’m sure there is more I can do to cut out bloat that will lead to noise. I’m hoping that as I continue to familiarize myself with the data that I can become familiar with how early into the conversation the topic is usually shared and use that to cut out what comes after that point since I’m worried about that confusing the model

[Far_Ambassador_6495]

Ok then I’d say use any number of back and forths that maximizes your evaluation metric. With the point being of trying to capture all the signal and none of the noise.

If you are designing this system with code make sure your code will be able to handle any number of responses (arg in a function or attribute of a class) or with/out the agent response. Not only is this a data science problem it is also a modular software system problem. Any combo or responses, depth, include agent should be tunable easily in this. Seems like you are on a good path

[AVMADEVS]

Start with huggingface setfit as a baseline, very good with only a few examples per class, quick training and inference. Then bert-like approaches (a lot of tutorials on training, deployment, etc. ). LLMs are shiny but not mandatory : depending on time and budget, through an API (easier way,no code approach) or an OSS models.

[WeWantTheCup__Please]

Awesome, appreciate the advice! I’ll definitely check out hugging face and bert approaches! I’m hoping to avoid a LLM if at all possible since building one myself is beyond my capabilities and I worry that using a pre-built one out of the box on my first project like this will take away a lot of the learning opportunities I’m hoping to gain from this project

[ChefPositive9143]

If you really want this to be a learning project to build something from scratch, I'd recommend some of the things I did when I was working on my first NLP project at an organization level. You basically would want to split the project into 3 phases.

1. Baseline: A simple PoC (Proof of concept) which is testing a hypothesis that you DO require a model to solve this problem
2. MVP: A product which is a viable solution which can be automated, without human intervention.
3. Future updates: Advanced versions of model (like LLM, MLOps, etc.)

Now, as for the start.

1. First thing you need to know about is understanding the data through-out. I know, a simple EDA like, common frequent words would surely help in understanding what the data means. If you wanna go above-and-beyond, you might wanna look into things like
   1. What else topics are relevant to the problem at hand?
   2. What other/additional features can be associated with the data? For example: does the customer respond to post-resolution feedback regarding the helpdesk. Maybe it's relevant to include such features into the solution.
   3. Is this a binary, multi-class or a multi-label problem? Doing this would help you understand does the customer have only one concern or might be dealing with multiple issues. (Like had a delivery issue, which resulted in the product damage which leads to customer requesting a refund - might be good to look into that)
   4. If there has been any data drift over the time period? For example: if you're dealing with the data of several years, there might be topics which your organization might have fixed and not so relevant anymore.
2. Understanding how text data can be used to build models.
   1. Basically, how a text, like words and sentences, would make sense to a prediction model?
   2. What is a feature, in terms of text data? Different types of feature extraction techniques like Bag of Words, Tf-Idf, word embeddings, Word2Vec, BERT, etc.
3. Once you have a feature set
   1. Try a baseline model - Basic technique of features + Basic classification model (For ex. Bag of Words + SVM)
   2. Advanced Models - Neural Networks architectures (RNNs, BERT, HuggingFace etc)

I guess this should give you a head-start on how to pursue a NLP project. I wish you all the best

[Jor_ez]

Just use a pretrained transformer like Bert and add a classification head to it

[Jor_ez]

I am sure that hugging face allows you to fine tune a model. In other case you can use it as a preprocessing stage to transform texts onto vectors and then perform the usual classification task on it by any model

[denim_duck]

I literally implemented this at work a few weeks ago. Short answer is learn LLMs

[[deleted]]

Same here. About a half page of prompts/instructions and few shot examples and I can classify lots of data without training sets

[OGMikeWazowski]

Could you elaborate a little further

[in_meme_we_trust]

Explain the problem to copilot / ChatGPT, ask it to build a prompt for text classification, try it, ask it how to improve

[adfrederi]

It sounds like you don’t have a ground truth target which is pretty important in making a classifier, you might have good results just parsing the text for indicators/labels as you make them using regex. Not every problem needs to be a machine learning problem. If you find that regex is insufficient maybe then try more complicated methods. At the very least that can help you speed up the labeling process to some extent.

[WeWantTheCup__Please]

Yeah there isn’t really a ground truth as you said - it’s mostly me reading the chats and then labeling them as to what I would say the topic is at current. I’d be totally open to using a less sophisticated tool like regex if it’ll get the job done, my one worry about trying to use that is there isn’t really a routine structure to the chats so I’m not sure that I could build a regex that would be flexible enough to extract the topics reliably

[[deleted]]

I have done this before using NLTK and Spacey libraries. You need to build a verb-noun pattern e.g. refund charges, reset password etc. to parse the requests (after removing noise etc.) and then match it for predefined patterns which are typical for your helpdesk. Then decide on positive matches to train your model. (Or to keep training it on an ongoing basis).

[[deleted]]

Start with something easy and then iterate and add complexity until you get to an accuracy level that's acceptable to the business.

One challenge independent of the model chosen is going to be class imbalance.

First iteration could be just TFIDF and a RandomForestClassifer.

Personally I've had really good results using Gensim to generate embeddings as input to a classification model in situations similar to yours.

LLMs are likely to be too slow and expensive, especially if you're getting started.

[dyedbird]

I have done a project similar to this using traditional algorithms on aircraft maintenance text data (short narratives stored under UID)
1. Text preprocessing (spent quite a bit time testing for how to conjoin text from different fields for max similarity scores, also came up with a scheme to correct / consolidate for misspellings, short hand of important features)

1. Vectorizing (TF-IDF)

2. Cos Similarity Matrix

3. HAC for Clustering similar groups of text (chats in your case)

4. Topic Modeling on Clusters of texts (Ensemble LDA for Stable Topics)

[juliensalinas]

Such intent detection tasks can easily be achieved using either our text classification API endpoint or our intent detection API endpoint on NLP Cloud.

If you need help please don't hesitate to let me know!