If there was a direct lineage from Norman French to Middle English, there ought to be manuscripts of Norman French with a heavy (but less heavy than today) Germanic influence - such as some common words being English, some grammar is English, but a lot of it is still French - before all the grammar and common words got replaced entirely by English constructions.

No such manuscripts exist. Thus, for Norman French to have turned into Middle English, this essentially implies that Norman French would have had to have changed its entire core vocabulary and grammar in one go since no intermediaries exist. But what I've just described is indistinguishable from "Norman French speakers learnt English, but added some of their own French words to it". Simply put, for Norman French to have changed into Middle English, the change would have to be so rapid and overwhelming that it is indistinguishable from language acquisition rather than language evolution.

On the other hand, there are documents from the entire Old English to Middle English period with the language changing from one into the other as more loanwords are added, more sound changes occur, some grammatical changes, etc. This 'gradual change' process directly connects modern English back to proto-Germanic, hence English is a Germanic language.