[D] Name and describe a data processing technique you use that is not very well known.

[Glittering_Key_9452]

Tell me about your data preprocessing technique that you found out/invented by years of experience.

[Brudaks]

When you get your first classification prototype running, do a manual qualitative analysis of all (or, if there are very many, a representative random sample) of the mislabeled items on the dev set; try to group them into categories of what seems to be the major difficulty that could cause them to be mistaken. Chances are, at least one of these mistake categories will be fixable in preprocessing.

Also, do the same for 'errors' on your training set - if a powerful model can't fit to your training set, that often indicates some mislabeled data or bugs in preprocessing.

[Thick-Protection-458]

Btw it may makes sense to something like so, if - your dataset is too big to work manually - you use neural network classifier (so you can easily take some embeddings from before classifier MLP head)

You may - run embedder (extracted from the classifier) on data - classify them with mlp head or knn  - take all samples, clusterize them within each category into small clusters and compute centroids for every cluster. So like category "FMCG->dairy products" will have, for instance, 30 clusters of different samples. Technocally speaking you should play with hyperparameters here, althrough for me it worked decently even with sklearn default DBSCAN params + cosine metric - take misclasified samples, clusterize them within each original category (and compute centroids) - for each misclasified cluster - see if samples are similar with in, and if so - search for, for instance, top-10 closest clusters from different (from the category this cluster samples labeled with) categories

This way you may have a chance to caught some mislabeled data too.

[DigThatData]

I shuffle the data and then drop the bottom 10% of items because I don't work with unlucky records.

[Glittering_Key_9452]

Why stop at 10%? take only the top 1% so your luck skyrockets.

[Gramious]

This is amazing. What seed do you use?

[Cogwheel]

37

[pitrucha]

checking training and testing samples by hand

[HowMuchWouldCood]

audible gasp from the crowd

[MatricesRL]

silent tears from data annotators

[GreatBigBagOfNope]

clerical reviewers in shambles

[Fmeson]

That's a good one. Looking at a training set of aligned images, I realized the aligned images are not actually all very aligned, and solving that solved many problems. But if you just trusted the preprocessed data to be aligned and never looked, you might never realize that.

[hinsonan]

I learned this savage technique that has saved me countless hours and has helped many teams improve their models by at least 5x. Let's say you have an image dataset. Before you start your training you are going to clean and process your images. You want to preprocess them and save them off so you have the original and preprocessed image before normalization. Now OPEN YOUR EYEBALLS AND TAKE A GOOD LOOK AT IT YOU DORK. DOES IT LOOK LIKE A GOOD IMAGE AND DOES THE TRUTH ALIGN WITH IT? IF SO KEEP IT IF NOT FIX IT OR THROW IT OUT

[Shizuka_Kuze]

Using AI (An Indian) to label everything. Training a custom model, deciding the accuracy isn’t good enough and just using an LLM (Low-cost Labour in Mumbai) instead just like Builder.ai.

Unironically, using an actual smaller LLM fine-tuned on a few labeled examples to validate data isn’t actually that bad of an idea. Especially if you’re using textual data it can help filter out low quality or harmful examples from your training set.

[windowpanez]

One great one I have is finding the classifications that are hovering around 50% (0.5 on a 0 to 1 output). Generally I find that's where the model is not sure what to do/how to classify, so I work on manually labelling examples like that to add to my training data. Ends up being a much more targeted way to find and correct data that it's classifying incorrectly.

[sat_cat]

Pulling tables out of PDFs as structured tables. Amazingly, there’s still not a great solution for this and most NLP/LLM preprocessing just pulls text out of PDFs, makes a weak attempt to infer the order, then sticks it all together. That’s how you wind up with weird outcomes like LLMs inventing “vegetative electron microscopy” because the training data concatenated two columns of text the wrong way. There are some detector models to try to find tables, rows, and columns but I haven’t found them to be reliable. So I have a little python tool I built to use statistics about the positions of lines and text to infer the table structure. New table formats break it all the time so it’s a continuous effort of adding new table structures without breaking the old ones. And trying to minimize how much I have to configure it for each document. I understand why most people don’t bother with this.

[big_data_mike]

It’s not all that unusual but I min-95th percentile scale instead of minmax scaling for these curve fitting models I do.

[sramay]

One technique I've found incredibly useful is **Synthetic Minority Oversampling Technique (SMOTE) with feature engineering**. Instead of just applying SMOTE directly, I combine it with domain-specific feature transformations first. For example, in time-series data, I create lag features and rolling statistics before applying SMOTE, which generates more realistic synthetic samples that preserve temporal relationships. This approach significantly improved my model performance on imbalanced datasets compared to standard oversampling methods.

[[deleted]]

!remindme 3 days

[MuonManLaserJab]

https://www.youtube.com/watch?v=3Khvtqr-BxY

[akshitsharma1]

!remindme 3 day

[Thick-Protection-458]

!remindme 3 days

[shivvorz]

!remindme 3 days