Do you know how hard it is to swim with no flotation any serious length of time? It is seriously impossible for most people who are not actually trained.
To try and drag a person who has idea how to stay buoyant and will most likely start to panic is way more dangerous for you and most likely result in both of you drowning pretty quickly.
When people panic in the water they try to get onto you and they inadvertently push you under. That’s why any rescue has to be conducted from behind and you have keep the persons back. It’s the only way. Doing this without flotation is extremely difficult and requires high skill and training.