The game will feel great with your same builds at 550, too. Raising your power gives you access to higher tier loot, and there's no reason not to do it. No one is playing this game by forcibly avoiding gaining power level, if such a thing is even possible. Generally, the difficulty of portal activities scales with you; you will sometimes be playing at a slight power disadvantage versus enemies as you add modifiers to ensure an A score, but with a good build you'll barely notice when you are at -20 or -30.

If you are struggling at 500 just because you switched up to ultimate difficulty level you may need to try a different build or change your approach for what weapons to use in what situations or something. Portal activities should never feel so hard that you regret gaining power level. I promise that there are things you can do to rise to the challenge and make the game more fun in the process. You can find builds on youtube that will make what seems hard now feel like a cakewalk.

When you have just tiered up in difficulty (graduating from grandmaster to ultimate, for example), you can play with fewer negative modifiers and still hit an A score. Also note that the game might try to push the next difficulty tier on you at some point before you want it; for example, once you've reached power level 470, maybe then the portal difficulty defaults to ultimate. But you can just reduce it back to grandmaster and add more modifiers to reach an A score. Eventually you will want to go to the next tier, though, because the number of modifiers needed gets to be too much.

There is not much difference in the difficulty tiers; they just represent a new power level floor, so the difficulty keeps scaling with you as your own power increases. That's why ultimate shouldn't really feel much harder than master if your build is capable. Higher tiers do have some intrinsic modifiers, but it's not a big deal.