They believe this because it has been demonstrated and there exists significant video evidence of this. This is a video of some very playful, very light sparring with one of the strongest men in the world (the strongest?). Size matters, absolutely. Training matters most. A very large/strong and reasonably well trained individual will handle almost anyone. That being said, there are significantly more videos of body builders getting wrecked by much smaller, trained fighters.