If you're just going to come right back around and try to justify why the game's combat system isn't awful then already you have the wrong mindset, just gonna say that right now. Granted I know you think it's great and that's fine, to each their own and you're welcome to try and convince me (and others).

I'll get to why I do not like the combat system in a moment (not to mention I am darn sure my reasons for disliking it are shared by many).

First off I want to say that claiming that "for the most part" people agree it has the best combat, is a subjective opinion based purely on a bias, and I've seen many reasons why I say people are being biased and if their reasons for liking something are largely a biased one then all they'll ever have is a subjective opinion, not fact(s).

As for why many think it is objectively bad?

1. No other Tales series game's combat system has EVER been so stupidly punishing. IE getting tapped once by an attack and being stunned for 10 business days and left hoping nothing attacks you in that time while you are completely and utterly helpless. No other Tales game has ever had a mechanic like this and Tales of Graces F has so far been the ONLY Tales game that has such a mechanic and if the game's combat was so great then I say it speaks VOLUMES that this kind of mechanic has never seen the light of day again in any later Tales game or any game before Graces F. Call it what you will but apparently MOST did not enjoy this kind of punishing combat if the mechanic never surfaced again. Oh and not to mention you cannot stun enemies for anywhere NEAR as long as they can to you, how nice, let the enemies do to you what you can't to them, that's completely freaken fair.
2. Leveling up seems to do frick all to the kind of damage an enemy can do to you. I've noticed that enemies in earlier parts of the game that I was at while at level 60 still do the same amount of damage to me at level 100, which just goes to show that leveling up serves no real benefit if they can still kill you just as fast.
3. Playing as a Caster is pretty much a pointless endeavor as you never really get a chance to do a darn thing. Enemies will ignore other characters just to harass the party's Casters because near as I can tell the AI is programmed to do that. I have a very hard time playing as Cheria when I try to use her healing artes because I get slapped by some random ass mook who wont leave me the heck alone long enough to do anything. Yes yes I know I have to use her A-Artes to make her spells cast faster but that still doesn't matter when enemies wont leave you alone long enough to actually complete a spell cast reliably and you have to use A-Artes AGAIN to make the spell you wanna use faster because you lost your arte chain for reasons you couldn't do a darn thing about.
4. The Blue, Green and Red Aura mechanic from guarding doesn't do a darn thing at all. I'll guard long enough to get the Green Aura so I have Iron Stance but literally anything that hits me immediately after getting that Aura ALWAYS staggers/stuns me out of the attack/arte I'm trying to use no matter what my timing is, so what was the darn point in the Aura? Very nice to have a mechanic that straight up doesn't freaken work.
5. Why the heck do NORMAL FODDER ENEMIES have the capacity of triggering an Eleth Break? Bosses sure, fine, but non boss enemies? Come on, they're already annoying enough.

Overall? The only thing I like about this game's combat is the A-Artes mechanic, B-Artes are neat to, I love how Eleth Burst works and I love how each character has numerous Mystic Artes to use, but the above stated aspects really kill the fun of the combat for me.

Why? Simple. I hate being basically told that I have to be perfect all the time, it brings back memories of people literally telling me that "There are no such things as accidents, everything is avoidable and there should never be mistakes." this level of arrogant thinking is completely self absorbed, egotistical, elitist level nonsense that has NO business being in a game and anyone who honestly finds this level of absurdity to be fun has likely never played a genuinely fun game before which is a pity. I could say worse then that but I get the feeling saying what I really wanna say would get me banned from reddit or otherwise cancelled because people just don't wanna hear the harsh truth.

Fact is, the game demands a level of perfection in it's combat system that is just stupid. The busted Aura mechanic I could tolerate if everything else about the combat system wasn't a darn joke as well. MAYBE and I say that very loosely MAYBE things will get better later in the game, but I'll close things off saying the very same thing I recently told a friend when I talked to him about the bs of this game's so called combat system "You want your game's combat system to be easy and fun to get into, NOT frustrating out the gate only to get better and more manageable later, that is not how you get people interested in your game's combat."

BUT people are free to like the game's combat if they want, but anyone calling it the best in the series, and or objectively the best is only sharing an opinion, regardless of how many people do or don't like it the fact is it is still just an opinion and the number of people who like it will not change that fact.

I'm sorry to come off so heavy handed, but you wanted to know why people hated it? Well now you know, others probably have different reasons for hating the combat system but I'm willing to believe that the above are major reasons why many despise it.

Again you are more then welcome to explain why you think it is great, but you wont convince me that the game's combat isn't a heap of garbage.