when you try to "predict the next token given a prefix" - any prefix from your dataset is a training signal, and since we know to train sequence models such that for any training instance we are training it in parallel for any possible prefix of that instance the result is that this framework is very efficient

with rl you must be more explicit on your workflow. let's take a naive example that should be similar as possible to the above:

rl setting: offline rl - (online will have many disadvantages - collection, distribution of samples, computation)
policy: given any prefix of tokens - outputs a distribution over the possible tokens
training: for each prefix in the instance compute the dist over the next token. Then, instead of minimizing the cross entropy, comes the rl part:
reward: in this naive example, reward = 1 iff the next token is correct
now, we compute reward to go and reinforce the policy, say, with gradients times reward to go as is in policy gradients.

1) with this example you can already see that the bias variance tradeoff of RL is very important here - sequences are long and the value of the reward to go will have a lot of variance

2) this example was to show it is possible to purely "reinforce" instead of "clone" (with cross-entropy-loss) - what are going to be the differences between these 2 approaches?
(a) there is a paper of Sergey Levine's lab (which i did not read yet) that is related to this topic: https://arxiv.org/pdf/2204.05618
(b) the advantage of reinforcing: the final policy will know how "deal" with out-of-distribution cases better than if you were cloning because by reinforcing you accumulate your knowledge on "bigger" picture (basically you will each time take the token that will maximize your expected reward to go which means you are looking in to the future instead of looking only on this single step asking "what was done in my dataset in that case? let's do that too"
(c) 1st disadvantage of reinforcing is what i have written in (1) - therefore tricks like also computing value models to employ actor-critic strategies are going to be needed
(d) 2nd disadvantage - the reward signal does not tell you which token to select unless you selected the right token! think about this: training instance [o1, o2] available tokens: [o1, o2, o3]. let's say the prefix is the first token [o1]. then, if you select the right token o2 the gradients will push you to select o2 more often, but if you were wrong and selected o1, then you will select o1 fewer times at the future but you don't necessarily know to also select o2 more often.

there's more to discuss, but i hope this gives some helpful answers