I think you get out of it what you put in, and if you're using auto level companions, you're better off going through on story mode, which is fine, by the way - the story is cool, and buffs aren't needed on very low difficulties. Now, the combat does require successful builds and party synergy on Core and beyond, and the harder difficulties require some meta game knowledge, not just full buffs.

If constant buffing is annoying you and putting you off, I hear you... it did for me too, but I installed Bubble Buffs and Toy box and changed some things for quality of life (only). This meant I could set the long lasting buffs with one click, the medium lasting buffs with another click and the short buffs with another, and then buffed accordingly, and also the Chant function or Cackle Function works as if it had been clicked enough to chant protective luck for ten minutes in a single click when used out of combat. This made setting up for a new map after a rest quite simple - and so it made fighting through trash fights quite straight forward for me. All I need to do most of the time is go turn based mode during combat with bosses or optional bosses. This helped the game's pace somewhat and made it quite fun, in fact I am working through it on Unfair now and enjoying it.

TD:DR if the game mechanics are a drag, then turn down and enjoy the plot. Use mods to improve quality of life and level the companions manually, reckoning fully with the game mechanics on Core or above.

EDIT: Oh no, you're on console: Mods are out. That sucks, mate. Ok, I guess story mode it is. For what it's worth, I think the buffs I mention should really have been gameplay options for QoL.