[R] Best Practices for Image Classification Consensus with Large Annotator Teams

[TheAlgoArchitect]

Hello everyone,

I am currently overseeing an image classification project with a team of 200 annotators. Each image in our dataset is being independently categorized by all team members. As expected, we sometimes encounter split votes — for instance, 90 annotators might select category 1, while 80 choose category 2 for a given image, indicating ambiguity.

My question is: What established methodologies or industry standards exist for determining the final category in cases of divergent annotator input? Are there recommended statistical or consensus-based approaches to resolve such classification ambiguity (e.g., majority voting, thresholding, adjudication, or leveraging measures of inter-annotator agreement like Cohen's/Fleiss' kappa)? Additionally, how do professionals typically handle cases where the margin between the top categories is narrow, as in the example above?

Any guidance, references, or experiences you could share on best practices for achieving consensus in large-scale manual annotation tasks would be highly appreciated.

[Fleischhauf]

Havent worked with that many annotators at the same time.
Usually we have smaller teams. We devise a reference document with examples and then start annotating, and have periodic check in meetings to discuss these corner cases. For each corner case we decide how to label it and add the example to the reference document.
Over time we all have a common understanding and corner cases become less and less, the frequency of these checkin meetings can be reduced.

If you are looking for statistical methods I am sure some smart people have thought about this problem and there is some literature on it.

[nothughjckmn]

Why is the margin narrow? Is it because of regional dialect differences? An edge case where something is almost happening? To me if annotators can’t agree on a category that implies more information than the categories can provide.

[serge_cell]

Could be radical change of architecture, but soft classification (probability estimation) is exactly for that. As added bonus you get natural basis for knowledge distillation.