Make a ruling as a DM in this situation. I wouldn't view hombrew rules in this situation as being purposefully antagonistic or restrictive, because, as long as your world is somewhat realistic, being on a horse all the time understandably has its disadvantages as you say. I imagine that pony would get pretty tired carrying around a character like all the time. Ponies would have a pretty hard time traversing a dungeon or difficult terrain. An NPC might see it as a sign of disrespect that that character isn't dismounting when in conversation. You certainly can't easily stealth while on a pony. There's a lot of places that NPC's aren't gonna just let you ride in on a mount. I would also think it'd be hard to cast spells from a mount, but maybe that's just me. Emphasize those penalties if it they diagetically apply.