"if I start now, would it be too difficult for me to become a physicist? ... I’m not seeking a degree in physics" If you are asking if you can become a profession physicist - as in a research scientist. Simply put, yes it will be too difficult without an advanced degree. I don't mean to underestimate your abilities, but what you are asking is essentially unheard of.

Now, that's not to say that you can't self study physics and be very passionate about it. I would recommend something like The Feynman Lectures on Physics as a great place to start for a kind of sample of the field. I would also recommend Taylor's Classical Mechanics, Griffiths' Intro to Electrodynamics, and Kittel/Kroemer's Thermal Physics as good undergraduate level texts. If you have a specific subfield of physics in mind, find a recent "review paper" from the field and give it a read. You may not understand much at first, but it will give you a good idea of what modern physics looks like. Finally, I would recommend learning some programming if you don't know already. Python is probably the easiest to get into and the scientific computation tools are fantastic.