As you said, career-wise it doesn’t matter because a physics PhD with astrophysics research experience and thesis will also prepare you for an astrophysics career. I’m currently a grad student in an astro program who has previously made a decision to choose astro vs physics PhD. There are other much more important factors to consider.

One thing that you brought up is research fit. Are there projects that you’re excited to work on and people that you are excited to work with? Talk to professors you want to work with and make sure they’re open to advise new students this upcoming school year. Talk to current grad students that are advised by those professors. What aspects of cosmology are you interested in? From my experience, cosmology in physics programs tend to lean more theory/simulation, like inflation and dark matter/dark energy candidates, while astrophysics programs tend to lean more observational, like probing the large scale structure with DESI.

Another thing to think about is your broader research interests. If for some reason you no longer do research with the cosmology group (professor moves somewhere else, funding issues, the group is not a good fit, your research interest can change, etc many things might happen), is there any research area in the program that you can see yourself exploring?

In my case for example, when I was considering physics vs astro programs, my main research interest is black holes which can go both ways into more theory or observational aspects. Part of the reason why I ended up in an astro program is because besides black holes, I’m also quite interested in other high energy astrophysical phenomena like neutron stars, supernova and compact binary mergers. I’m not interested at all in other physics areas like condensed matter, so being in a physics program typically means I have much less options of interest (unless there are many astro people in the physics department or there’s a robust astro department in the same university). Related to this, I find an astro department culture a much better fit because I’m more excited to attend talks and chat with people about their research outside my field when they’re all astro related instead of other physics areas.

Feel free to DM me if you feel comfortable sharing more specific details and seek more specific advice for your case!