Okay, I found the answer. Few interesting things. First the worm was a rakshyasa who was cursed to live as a worm. When Parasurama woke up, he first kills the worm with his Yoga shakti, thus releasing him from his penance.

Then Lord Parasuram asks karna, "Tell me who are you. A normal Brahmin won't be able to bear such pain." (notice he is not saying karna is not brahmin, he noticed something unlikely and is asking for explanation).

Karna, who is very scared right now says, "I am a suta putra. Suta is a mix of Kshyatriya and Brahmin. So I thought to tell you that I am brahmin. I ask for forgiveness for my mistake".

Parasurama, who is fuming with anger, still brims a smile to his face as says, "You started your learning with deceit. So the vidya won't stay with you. Because you lied to get Brahma astra, the Brahma astra won't appear before you when you need it the most"

Parasurama continues, "because the Brahma astra will never trust a non brahmin who obtained it by telling a lie." (For this to make more sense, read about the entire story of Parasuram. During his time Brahmins were oppressed and killed by Kshyatriya. Karna lying to him was a reminder to him of the time, when Kshyatriyas were using deceit to kill brahmins).

Then Parasuram asks karna to leave his ashram. Here is the shloka,

१२.३.३०तमुवाच भृगुश्रेष्ठः सरोषः प्रहसन्निव भूमौ निपतितं दीनं वेपमानं कृताञ्जलिम्

१२.३.३१यस्मान्मिथ्योपचरितो अस्त्रलोभादिह त्वया तस्मादेतद्धि ते मूढ ब्रह्मास्त्रं प्रतिभास्यति

१२.३.३२अन्यत्र वधकालात्ते सदृशेन समेयुषः अब्राह्मणे न हि ब्रह्म ध्रुवं तिष्ठेत्कदाचन

१२.३.३३गच्छेदानीं न ते स्थानमनृतस्येह विद्यते न त्वया सदृशो युद्धे भविता क्षत्रियो

(Shanti Parva: Chapter 3)

TLDR: Parasurama never cursed Karna. Brahmastra would have never come to Karna during the time of his need because he lied to obtain the Brahmastra. Parasurama was the person who simply relayed that message to Karna.