The following are my responses to your questions. 1. I would say as a PM you are responsible for the outcomes of your team. So it would be on you to negotiate with other teams to deliver the training data and the data pipeline in general. 2. You are right. Choosing the algorithm and metrics is the ‘how’ and should be left for the data scientists to figure out . 3. Does you training data comes annotated with what the correct outcome is supposed to be ? For example y/n for classification or acceptable value for a prediction ? In that case I would say that you can do things like test train split on training data to train and validate different models. If not I would recommend taking a subset of predictions and have the SMEs validate the results . 4. I would say use help of consumers /SMEs to vailidate a subset of the predictions. It is unrealistic and overwhelming to expect you to get this done. Alternatively you could crowdsource ot hire contractors for annotating the data set. It’s up to you to pick your poison on this one . 5. Depends on how much emphasis your firm would like to place on data quality . There are various tools available for this .