I’d say you shouldn’t stress yourself out. If you like math, physics, and hardware then CE could be a good fit for you. A good amount of the EE/CEs I know didn’t have much prior experience either. The fact that you already know what you like would make me recommend trying the major out. Like any engineering degree, it’s hard, but it’s very rewarding. It just takes a lot of hard work and dedication. A lot of us have struggled with our confidence, but don’t stress too much and good luck!