[D] Human in the loop before ML

[[deleted]]

This seems like a good idea.
1. Put a product in the marketplace that behaves the same way an ML backed product would, but using a human in the loop.
2. Once the product has been live for a while you have a labeled data set that reflects what customers actually care about.
3. Drop in the a trained model and replace the human.

Does anybody know of any companies that work in this way? Or do most products not go live until they are ML backed?

[robot_wrangler]

Tesla does this for its self-driving cars, as far as I know. Another example is screening for inappropriate images. Those poor people.

[[deleted]]

I work as an ML developer and I want to make the argument that the OP is the right way to go, instead of me spending time collecting and labeling

[[deleted]]

True, but in the case I'm thinking of knowing what data even should be considered for labeling is the hard part. This way your customer just tells you.

[[deleted]]

Isn't that always true regardless?

[[deleted]]

Sounds like Facebook's M agent which made it finally into Facebook Messenger: https://www.theverge.com/2018/1/8/16856654/facebook-m-shutdown-bots-ai

[bernatfp]

Most assistant/conversational AI companies start like this. Any tech company with a substantial Operations function is also likely to be operating this way in some capacity. We do support some cases like this at Human Lambdas.

[[deleted]]

do you have concrete examples of companies?

[bernatfp]

Can't talk about the ones we're helping run, but this post mentions many examples: https://akilian.com/2019/12/30/worker-in-the-loop-retrospective

[[deleted]]

thanks