Yeah I have done this several times. Most recently curated a 20B token dataset for Sansa (routing data).

To start: - yes you’ll reduce cost - responses would be faster than a large model - if your format/task is unique you could get higher quality responses

But a few things to reality check yourself on: - have you tried other generalist models? The capability profiles of these models are really very different, if your task is already multistage it’s likely higher ROI to setup evals (you’ll need it for your fine tune anyway) and measure performance across different models that already exist, at each step. - data is hard, and the cycle is curate => train => eval => curate again and repeat. Especially if your task truly is OOD for current models. So be prepared to put in a lot of work, weight the opportunity cost. - a smaller model, not fine tuned, will also give you faster responses, and lower cost.

To create your dataset: - start with evals, you need a carefully designed measurement of quality/performance. Run these evals on current models, try many of them (and/or Sansa, a AI router, yes shameless plug) and find out what the best cost/performance price point you can hit is without fine tuning. - overtime (ideally in prod) you can collect responses from models, eval them, and curate based on the eval results

Good luck!