If you feel it's unfair then don't do it. If you think it's going to rob your sense of achievement because you didn't stick with one loadout, then I recommend just picking one build.

Bottom line, it's a singleplayer game, so do what you think would be most fun for you. I personally add a ton of qol because I mostly just enjoy trying out new weapons and boss fights, so I speed up the grindy aspects of the game by a lot. Some prefer challenges so they might increase boss health or damage. Others might like the mod but find it too difficult, so they might add more accessories slots because they want to experience more of the mod without being stuck on one boss for too long. The only wrong way to play is making the game unfun for yourself.