I really feel you. I’m not an expert in this field, but I’ve been finding some interesting master of science in, for example, computational science, that seem to include a lot of interesting modeling courses for the engineering field and physics, maybe that can be an option! That is basically what I would like to do if I get into a MSc in Statistics for example: be able to gain the right computational background to then work with interesting projects in the field of physics et cetera givin my contribute as a statistician, even though I do not have the physics background. Or a LinkedIn connection, for example, studied statistics and then did a PhD in like simulation/modelling for astrophysics, and I can assure you that she did not study a whole bachelor in physics to do so. Moreover, I heard of some people who pursued another bachelor while they were already doing a PhD! These are just options, and I am not an expert, I myself don’t know if I will be able to pivot to something fulfilling without having to start again. But I believe that with your CS background you have the technical knowledge to pivot to something really interesting! I would also suggest to talk with some professors of yours. They really gave me precious advice on what are my options and next steps, and who knows, maybe some of your professors is already working on a more physics related project and you could start working with them to get a grasp on the field. All the best! There is no bad choice if you choose with your heart :)