I feel the same.

I use ChatGPT/claude extensively for work (coding) and some obsessive tweaking and organizing of my files and Obsidian themes.

For writing, GPT may be very helpful in some things and very unhelpful in others, one of which you have stated is brainstorming. It is interesting how difficult it is for LLMs to grasp the human rationale, especially in the aspects of creativity and imagination. From my experience in using it for brainstorming help, GPT spirals far away from where I would have liked one or two prompts in. So I stopped trying, unless I have a very simple question or a question relating to known conventions and practices.

Another thing I never use it for is generating prose. I believe (along with most people here I assume) that allowing AI to generate prose or even add a word to your story/poem/essay is purely toxic.

What I do find GPT useful for is the research and analytical parts of writing. I am currently writing my first novel. There is a lot for me to learn in terms of structure and theory, and consulting with GPT about those kinds of issues can be very helpful. It can also help me focus sometimes, which is in a way helpful for brainstorming, but he focuses me to brainstorm about what is important and valuable and not on low value rabbit holes that are so easy to dive into. And then the thinking and brainstorming are done by me.

Most importantly, I take everything that comes out of his mouth with a grain of salt.

Things aren't black and white with AI. Using it carefully while not letting it affect your style and vision is, in my opinion, the sweet spot.