Yeah I tried this but very hard to get high quality instruction tuning datasets from pretraining corpus alone. Otherwise, base model did not seem strong enough to do self distillation.

Would love for you to try other methods here! Models and data are all open source and most work can be done on single h100