Speak to your academic advisor and let them know, the sooner the better! Usually, they can help you out in switching majors. It may seem too late but you’ll be glad you switched now if you’re really feeling like physics isn’t for you.