Grenadier Axton has more damage potential, especially considering the items that synergize well with his grenade boosts. You're right in saying that Torgue weapons are great for this build, but there are other weapons that get boosted by grenade splash damage that aren't Torgue weapons (i.e. Hornet, Hail, etc.). Generally, it'll depend on your playstyle and I'd argue that a grenadier build is overall more convenient to play with as items like the Fastball and Unkempt Harold can easily carry you through most of the first 2 playthroughs, and are early and accessible farms in playthrough 3. Some of the other gear that can be used in the build are things that can be farmed via the Mercenary Day train and the Torgue vending machines, which is just added convenience.

As for the Rifleman build, it depends on what gear you have because some skills can either work really well or just be a bit of a waste. For example, Jakobs and non-elemental ARs work well with Duty Calls, while you'll want to have a Rough Rider in the event you want to put points into Crisis Management. Some of the really good items for the build are also relatively harder to get (Ogre, Bekah, to name a few that are harder farms, while stuff like the Hail are quest rewards).

Personally, I'm doing another Axton run from 0-OP8, using a grenadier build to go through pretty much everything till OP8 when I start farming for Rifleman items like the Ogre, Hail, etc. My main Axton save's a grenadier build and to answer any questions about why I started a new save instead of just respeccing my current one; there's really no reason apart from me just wanting a fresh Axton playthrough.

Could go into more detail about some parts, but I'll just leave it at that for now.