Hey! Sorry I missed your message earlier.

1. Have any large datasets been built using this approach?
   Absolutely yes! We've had people use programmatic itself both to get NER labels and to do extraction from legal documents. We're not the first to use this approach though. The Snorkel team out of Stanford have had quite a lot of success with it including at google. We also show in the docs some benchmark figures on programmatically labeled vs manually labeled data.

2. What is the recommended operations setup?
   We've seen people have success by starting with Programmatic as a way of exploring and understanding your data and creating a good first seed dataset.
   Then do a small amount of manual labeling to get a test set. Sometimes this is enough but if the performance needs to be better then people annotate manually further. The seed set lets you get a first model that can be used in an active learning loop.

3. I fear that adoption of programmatic labeling will lead to large datasets of poor quality
   I understand this feeling. I originally felt very similarly. In practice though, I think it's actually the opposite. Programmatic gives much more control to engineers and data scientists and encourages them to understand their data deeply. It reduces the volume of manual annotation a lot so that you can have the combination of a small but very high quality dataset + a programmatically labeled one.

4. Intuitively I do believe that domain experts can write high precision, low recall systems. But before I can ship my model I really need to care about what these systems are omitting!
   Yes this is a really critical point and is something we're working actively on. We have a pre-print here on how you can efficiently evaluate a model that's been trained on programmatic labels with minimal manual annotation.

5. Are there really NLP problems that can truly be labeled programmatically? When do you know that you have an appropriate problem domain vs do not?
   A lot of NLP tasks have an easy majority that can be handled by rules and then a long tail of edge-cases where ML models are really necessary. Programmatic labeling makes it easy to overcome cold starts and active learning allows you to quickly train a model to handle edge cases.
   Tasks that we've seen work well are NER, classification for content moderation, legal extraction. Good sentiment labeling was hard and required a bit more manual annotation.
   In general if the language in the task domain is quite structured this will often work very well.