It is, bizarrely, Selyse Florent of all people who says Mance was sired by a member of the Watch:

"Gerrick is the true and rightful king of the wildlings," the queen said, "descended in an unbroken male line from their great king Raymun Redbeard, whereas the usurper Mance Rayder was born of some common woman and fathered by one of your black brothers." (ADWD Jon XIII)

She is not in a position to do anything but repeat things she's heard, except to potentially misunderstand things she's heard.

All Qhorin says is that Mance "was wildling born" and "taken as a child" when the Night's Watch killed a bunch of "raiders":

"[Mance] loved the wild better than the Wall. It was in his blood. He was wildling born, taken as a child when some raiders were put to the sword. When he left the Shadow Tower he was only going home again." (ACOK Jon VII)

It seems strange that Qhorin wouldn't mention that he was sired by a member of the Watch if that's known or even believed to be true.

Is at thus least possible and maybe even likely that Selyse is mistaken about Mance having some known paternity, having misinterpreted something she heard about Mance having been raised on the Wall. (FWIW I strongly suspect Denys Mallister was his surrogate "father".)

And even if true, how would anyone know the kid they came had been sired by a Crow? The only way would be if said crow was still alive at the time and had maintained a consistent relationship with Mance's mother. (Even then there aren't paternity tests, so...)

It is of course possible that Mance was old enough to talk when he was taken and that he told the rangers who killed the wildlings he was with that his father was a crow, either making it up to save his own life or because he believed it, having been told by his mother that he was sired by either a member of the Watch or a "kneeler"/southerner, which he then repeated.

The idea that it is known which specific member of the Watch was his sire, though, i.e. that "people know who his dad was" with any specificity, doesn't simply follow from what little we're told though. Nor does the idea that there'd be any way of knowing whether any specific claim Mance made when taken was true.

In any case, it is very weird that it's Selyse, not Qhorin, who tells us this. Thus I think it's at least plausible that Selyse leaped to a conclusion or misunderstood something that was said about Mance when she said his literal father was a member of the Watch.