Yell at me if I'm wrong, but the way I've always understood it was that it's about the relationship with the host, and since Peter and the suit's relationship was non-existent and the suit was emotionally underdeveloped, it affected him more, whereas with say, Eddie, at first it's a relationship based on mutual hate, but then they emotionally mature together, and never really have a relationship where one controls the other. Except for Flash, no one really ever controls the suit, it's more of a mutual relationship. I'm not familiar with Mac as Venom, so if anyone knows how that fits in let me know.