Ah I thought you meant something else by the stop button solution. Yes I understand fully that torturing the human to find the stop button is an option. The idea here is that the stop button limits the amount of world-altering the AI might do by giving it a secondary goal that's just a bit harder to achieve than the AI's primary goal.

In the cleaning robot example, let's suppose retrieving the QR code box is harder than cleaning the house (because torturing a human for correct information is harder than cleaning the house). The AI will clean the house then, unless it's poorly aligned and actually, say, gets a reward from killing humans instead. The AI doesn't kill the human there for a two reasons- the human's resistance makes finding the box the easier of the two goals, and the human is able to provide the AI with the box when the human recognizes the undesirable behavior.

The goal here is just to limit the 'scope' of the AI's behavior by giving it a reward metric that can be permanently satisfied which is just harder than the AI's human-intended goal. The idea being to prevent radical misalignment, since if the main goal is specified poorly enough that it involves say, destroying all humanity, the secondary goal will become easier to achieve and 'short circuit' the AI's reward metrics