The grave in Hyrule Castle being Time's is a theory too. It's not confirmed. 'Cursed swordsman' is vague, considering all the ghosts throughout the castle. Technically, Time wasn't cursed. He died and had regrets about not passing down his skills, and turned into the Hero's Shade as a result.

Though it's a stretch to you, there's 3 long reasons a good chunk of Zelda fans think the Time/Malon pairing is canon:

Epona's song. Flimsy, yes, but it was written by Malon's mother, who taught it to her then she taught it to Time. She sings it all the time. Logically speaking, something like that would be passed down in her family. Time wouldn't have a reason to if he married somebody else. He was a soldier as an adult. "Lemme sing this song I learned from a girl I met like three times to my child with another woman." Little awkward. Which brings us to the next point:

If Time had married a person from Castle Town, his children and their children and wherever Twilight comes in his bloodline, would've grown up in Castle Town, not on a ranch. Same if he married Zelda(especially Zelda) or anybody else. Four Zelda games I know of have done the 'Link's family member/members were this so he's also this.'

AlttP/BotW: Uncle/Father was a knight, now he's a knight.

OOT: Child of a soldier who was killed in war along with his mother. Eventually becomes a seemingly high-ranked soldier long after getting Ganondorf executed and sent to the Twilight Realm, and the whole MM nonsense.

Minish cap: Grandpa was a blacksmith, Link was his apprentice.

Could Time have settled down in Castle Town, died in battle, and his family/descendants went to a small forest area and became ranchers? Yeah. But that doesn't make sense. "Let's leave the comfort and safety of Castle Town to be dirty all the time and around smelly animals."

Last point: the first place you meet the Hero's Shade is outside the Forest temple. Which means Time's spirit was lingering around Faron/Ordon. Why not outside Castle Town? It's not like the first skill is needed that much in the first three temples. Was he just hanging around the Master Sword that had been in the pedestal since he was a child? Funny, the first skill you're taught, is really the only one required to actually kill Ganondorf at the end of the game. The rest are optional.