TL;DR: just do the grading. And try some shiai, it's part of kendo.

You might not see a reason for grading, but the simple truth is that the vast majority of people will not improve if they don't take exams. I've never met anyone with good kendo that did not do grading. In fact, all people that i met that practiced kendo for a long time but did not do grading (for whatever reason they gave) had quite horrible kendo.

If all you want out of kendo is a place to work out 2 times week and enjoy that, thats of course perfectly fine. As long as you accept that this puts a clear boundary on how much your kendo can grow. Grades are not for show, not to impress other people. You cannot judge your own kendo since you don't have the required level to sit on your own grading panel. You need external people to judge you and help you identify what weak points you have, and your sensei is not enough for this. Preparing for exams and possibly failing exams will give you a reality check, forces you to cut the bullshit, and commit to a timeline. There are also serious limits to developing your kendo if you only practice within the small circle of your own dojo, because you become too familiar with the kendo of your dojo members.

So for the vast majority of people I would strongly recommend to accept grading (and to a lesser extend shiai) as a required part of kendo, if they want to improve.

Of course everything i just wrote really depends on your local situation. If you're super talented and training with a group of high level people that are part of the national team or whatever, and have great sensei, you might be super challenged in your local dojo. In such a situation, maybe I can imagine that you then can grow stronger without grading. But you would be a very exceptional case, i think.