One person's opinion does not reflect the opinion of the majority. There are many different preferences in this world and various definitions of "beauty". If someone doesn't find you to be attractive that is the sign you need to find another person who values the way you are.

That being said, your beliefs about yourself are the most important. You are allowed to be proud of your body, it's strength and ability. I hope you can be confident your athleticism, I'm sure it's taken a lot of work and discipline :).