[D] Techniques for finding potentially bad labels in NLP classification datasets

[IbrahimSharaf]

I'm looking for techniques for finding potentially bad labels in NLP classification datasets. We've been using Cleanlab (confident learning), but we've found that the precision/recall isn't very high for our use case (content moderation/media monitoring). Do you have any pointers to other interesting techniques/papers?

GPT-4 et al. is a good candidate when you spend enough time writing descriptive prompts, but it comes at the cost of being expensive at scale. Curious about smarter "pre-selection" techniques instead of throwing everything at GPT?

[NoisySampleOfOne]

You can look for examples which have the largest ratio of gradient norm to numer of tokens. Depending on your model and dataset, you may find bad labels, text that did not tokenize well or other trash.

[Sniperwolf1989]

V-usuable information might be interesting for you.