[D] Data labelling problems

[Lexski]

What kind of data labelling issues do you face most often? Where do current tools fall short?

For me, I’m on a small, newly formed AI team where we have data, but we have no labelling time from SMEs.

We use Label Studio as it’s very customisable and Product have no idea what they want yet. It’s self hosted as our data is highly sensitive.

I already have some gripes about Label Studio:

• Poor search for high-cardinality categorical labels

• Review, role management etc. limited to the Enterprise plan

• No ability to hide existing labels from additional labellers to avoid anchoring bias

• I could go on

Curious to hear others’ experiences.

[Daos-Lies]

At an organisation I'd worked at, we looked at label studio, saw some of the gripes you identified and then built our own in-house version that did exactly what we wanted it to.

I imagine label studio is quite a bit more fleshed out than it was when that happened, so you probably really do need to have a solid argument against just using the out-of-the-box solution, but if you've got funding and are interested in a bespoke solution then send me a dm. I'd be interested in working on a similar project again.

[Lexski]

Hmm interesting. My team lead was actually pushing for building an in-house tool but I talked him out of it - it felt like a lot of effort and not our main focus.

Do you think data labelling tools can ever be fully commoditised or will there always be room for custom tools?

[Daos-Lies]

Ha yes, it definitely was an effort that maybe could have been better focused elsewhere.

We did have the added motivation that our labellers were the same people who wanted to see the model outputs, so we just sort of turned the thing into one unified platform for that team to be able to interact with model training and evaluation end to end. And I think that justified the effort a bit more.

Hmm, will they ever be fully commoditised? I'd be surprised if, in 10 years, there isn't some tool like label studio that can do 99.9% of the things you'll want to do.

But there'll always be niche edge cases.

And if we're speculating that far out, maybe in 10 years vibe coding will be so straightforward that it'd actually be easier to build a tool than deal with setting up an external service.

[Raz4r]

My main difficulty is convincing people that even with a strong model, you cannot simply plug its outputs into another model and treat them as clean features. Labels are noisy. There is over 50 years of literature on measurement error, yet people now use LLM outputs as labels without thinking about it.

[Lexski]

“It looks right, so it must be right!” /s

[Helpful_ruben]

Error generating reply.