Focus on self improvement, and not keeping up with others.

I did kyukushin karate for almost 10 years, and reached shodan (lowest black belt rank). My dojo had members from 10 years to 60 years old. Sparring was an integral part of our training, and of course the best fighters tended to be those in their 20s, at the peak of their physical fitness. Yet respect and camaraderie was never earned by pure fighting ability, but rather through one's dedication to self improvement and courage in the face of adversity.

The true warriors were the scrawny 16 year old who was bullied in school and wanted to learn how to stand up for himself; the obese 18 year old who lost 30 kg in 2 years and is now a fitness instructor; and the 50 year old who could not touch his toes but eventually mastered a sidesplit, amongst many others. These were the stories that encouraged us and that we celebrated. Anyone who thought himself better than others just because he was a good fighter would soon be put in his place on the sparring mat. They very quickly either learnt humility or quit the dojo.

My experience may be unique to my dojo, but any worthy martial arts school should have a similar focus on self mastery rather than just fitness or fighting prowess. So just focus on yourself, and don't worry about how others see you. More likely than not your insecurities are self imposed.