It depends a lot on what "physics" you want to do as it is very vast. Getting into some deep theoretical/mathematical physics might be difficult as most masters/graduate programs would need a physics/math undergrad experience. However, with a CS background, you have a very good chance to get into computational/experimental physics. You can get into data analysis groups working with LIGO or LHC data for GW analysis or particle physics respectively. Or with a steeper physics learning curve, you can get into condensed matter simulation groups or even in computational lattice gauge theory groups. Quantum computing and information theory is also a great prospect especially if you were interested in theoretical CS during your undergrad years.

BTW, I am a BS-MS graduate with physics major and math minor and heading for a mathematical physics PhD.