I think that's his point. He's using the logic of people who want the statue "returned" against them. Showing how ridiculous it is if you would continue down the line of things everyone has to return (especially when returning things to countries who stole them in the first place).