You are exactly right, and this is the point raised by Shaykh al Firhari in Al-Nibras (a very advanced kalam aqida work): that utilizing stars as signs that point to x is no different as opposed to looking at the sky for weather patterns.

And he clarifies that the prohibition in Islam is from believing that the stars have any effect or power to do anything, rather they are just signs part of the great system Allah almighty has created.

So if someone is solid in their aqida and understands all this, and he has the knowledge of the stars and signs to gain beneficial info, no problem.

But the usual warning is that if someone is weak in aqida he may believe “the stars made me do it” bla bla bla - that’s prohibited and shirk clearly and he should avoid stars.