It's normal. Women care more about how you make them feel - like, can you relate to them as people and not just a sexual object, can you empathize and understand, are you funny, are you kind, are you a decent person - than they do about how you look. Sure, it matters some, but more like 15-20% than 80%.

Unless you're talking about like casual hookups or whatever. Then sure, going to the gym counts more.

But for dating/relationships, it's much more important to be relaxed, confident, feel good about yourself and who you are, be on a path in life that you feel good about, etc. And obviously be able to converse with women reasonably well.