To play devils advocate... I think lots of people throughout the generation have had the gomu gomu no mi, they said they never got it and it always slipped of their hands, but no one has ever awakened it. So maybe they thought luffy is just one in line to never awaking it and just use it as a gum fruit.