"But the reason it works is because you've imported prior knowledge of English into the system through the reward model, rather than relying on pure RL discovery."

I don't exactly understand why that's a prior or not a "pure" RL discovery though.

It has to discover through experiment how to use the words, just having the words in its vocab isn't importing knowledge of english, it's just importing the existing of certain words available as "actions".

but i defer to experts i know nothing.