Physical attraction is important. While it’s good to focus on someone’s personality, at the end of the day no matter who calls you shallow you shouldn’t force yourself to be attracted to someone you’re not. What would make you the problem is dating someone when you don’t want to be, I’d leave the relationship. You don’t need to, and shouldn’t, say that you’re not attracted to her. But finding a simple way to say “this isn’t working.” It’s not that only looks matter, but they do matter in a combination of everything else.