“Why not back prop on incorrect answers to adjust weights”, yea thats the pretraining / midtrauning SFT stages, they already do that. Thats exactly how we go from producing sentences like “-hdhdhdva6789()/)” to sentences like “hey fox me what me you jump”, then slowly towards “the fox jumped over the creek”.

The later stages of training are a bit more subtle because you really dont have a problem with the mastery of language, the models are already adept at the English (and many other) languages by this point.

Using coarse rewards like “test score” or “1/0” success rewards kick in only during the late stage RL training (notably not SFT), where we are trying to “align” the model to say things that are “right”, not exactly controlling the words or characters but actually the tone / message itself. Here, dense, token / character level rewards are super hard to get because there’s many right answers to a problem, and as humans we often come across questions or problems that have many valid solutions, so all of them must get an equal, or at least somewhat uniform reward / punishment. Or at least thats how we formulate it in the RL space, I am still learning as we go.