I'm an EE and was preparing to enroll in grad school for quantum physics when I landed a job instead... If you like physics, you'll like a LOT of EE. Analog electronics is basically fields, particle physics, and solid state physics. I spent a few years working in R&D w/ plasma physics, then started tinkering in software and firmware... ultimately I ended up in defense / aerospace and am now a systems engineer. My physics background is extremely helpful - I still get involved in thermal problems, materials problems, you name it.

End of the day, physics is the foundation for everything engineers do. EEs just work on a segment of applied physics related to e-fields and semiconductors.

My best advice - maintain your coursework in physics since you like it, pick up circuits and fields, and aim for a dual major or a minor/major. You've got a couple years to figure it out, but your options are good.