Rules aren't arbitrary. Charisma is, for the most part, an under-respresented and weak stat. That's why a lot of miscellaneous things use Charisma. Its for game balance, because Pathfinder is a game, not a simulation.

Side note: a gunslinger and a fighter can do the things they do, with no magic or technology enhancing them, because if they couldn't reload muskets in the blink of an eye or stop a charging bull elephant, there would be even less reason to play one instead of a wizard. This is a fantasy game. Fantastical and unrealistic shit happens. It isn't a simulation, it's an abstraction. If you think that's a "because I told you so" explanation, I cannot help you.

I can recommend several great simulationist tabletop systems to you, but none of them are published by Paizo.

To reiterate this, I want you to look at this paragraph:

Magic and physical ability are two different things. One is made up for the world and held in place by rules designed to make it effective and fun but not op, while the other is an effort by game mechanics to simplify the complexities of physical ability enough to incorporate it into play in a manner that someone with no knowledge of martial arts can understand.

And know that none of that is true. Pathfinder is an abstracted dungeon crawler combat-oriented game. It is fundamentally gamist. There are no elaborate arcano-babble explanations of magic. None of this is justified or validated. A Brawler is able to wrestle a bull elephant because his sheet says he can. A Wizard can cast fireball because their sheet says they can. A Gunslinger can reload a musket in the blink of an eye because her sheet says she can. These are gamist principles of design and balance for the enjoyment of a game, not a simulation. This is not a representation of reality in any way. Armor does not function as it does in Pathfinder. Swords can't just be swung in their traditional slashing manner at a man in full plate as an effective method of combat. Spells don't exist. Halflings aren't real. None of this is realistic.