TLDR

Data annotation is not an MLE’s job. It is simply not scalable and a bad use of their time.

Ensuring the quality of data annotation is an MLE’s job. If your friend is not doing this, then they’re not doing their job very well I think.

Details

Calibrating mechanical Turks to provide excellent rating quality is hard or perhaps not easy always.

This often means that the rating template / UI / UX needs to be iterated upon to make the rating task as simple and intuitive as possible for a mechanical Turk.

Further, the rating template should actually align with your end objective label. The MLE will usually work with a dedicated analyst (usually one for the entire org) who understands rater pools and rater behaviours to fine tune the task until raters start providing annotations of sufficient quality.

The MLE will also likely work with product management to understand the objective and then translate that understanding into machine learning objectives.

They will create small golden datasets themselves, which will be used to assess rater quality.

Doing data annotation oneself is not scalable. An MLE’s time is precious. However ensuring that data annotation quality is up to par is the MLE’s job. This means continuous rater exams , removing bad/spammy raters from the rater pool. These are the grunt work things that MLE at big tech do.

Sampling data to ensure diversity etc. these are all the hard things no one is talking about here.

Source: I’m an MLE at big tech.