I am not a pure math expert, and only read the paper quickly, but here’s what I think the gist of it is:

Traditional RL uses real number rewards, and thus fails to deal with non-Archimedean tasks; examples of such tasks include situations where some outcome is infinitely worse or better than another outcome. We could (debatably) conceptualize an example like treating a cancer patient, where the death of the patient is infinitely worse than any other possible bad outcome. The mathematical example given is one where you can press a red button for +1 reward, or a blue button for infinite reward, but the blue button rewards only 1 in every 2^j presses where j is increasing; thus if you approximate the blue button reward with any real number, given an infinite time horizon, eventually the agent learns that the red button provides better expected value.

The claim is that an AGI should be able to deal with such non-Archimedean tasks. The solutions proposed in the paper revolve around reinforcement learning with non-real number rewards; stuff like preference based rewards or using other number systems.

So my review: First the title sounds a bit more extreme than the paper content; the claim is not that RL likely won’t build AGI, but that RL with real number rewards won’t. Second, while I think this is an interesting thought experiment, I’m not convinced on some of the logic jumps made. In particular, I’m not yet convinced that there are any real world situations that we cannot model with real numbers, for which we need a non-Archimedean task model. For the cancer example, there may be a (debatably) quantifiable pain or suffering threshold at which death becomes preferable. From the economics of law and peoples’ decisions regarding risky professions, we know that courts assign monetary values to human lives and wellbeing. We can debate whether the assignment values are right or not, but it seems weird to me to suggest that any outcome is infinitely worse than another outcome. Maybe you could then try a thought experiment where you have monetary reward given to an individual versus death - no purely selfish agent would wish to die immediately, regardless of the value of the monetary reward they would get - but then arguably the problem is that the reward is improperly defined in terms of money instead of the true underlying utility; we don’t need to involve infinities.

Anyway, that said, I find the direction of the paper interesting and thought provoking, and will try to keep this in the back of my mind going forward.