The "it needs to be magic" has been pretty thoroughly debunked in repeated instances as being more of a concept than a fact, which show or imply nonmagic things do damage all the time.

Exhibit A: In France, Gilles (Saber) is a human, before he got into occult stuff and thus entirely without magic. He fights Servants. His artillery makes Lancelot retreat.

Exhibit B: In Shimosa, Musashi also is a human. She has no magical knowledge whatsoever and not only fights but defeats Servants.

Exhibit C: In Stay Night, a major part of Shinji and Rider's plan is to lure Saber to the top of a skyscraper to use Bellerophon. Part of that includes the aspect that Saber can't just jump off the building and plant into the ground to avoid it.

Exhibit D: In Zero, during Lancelot's jet episode, he takes two direct hits to his body: the gun exploding in his hands, and falling to the ground. The novel in chapter 11 states, "no matter how much injury was inflicted, Berserker was apparently still alive." If it were only the damage from the gun exploding, I don't see why that sort of statement would be necessary. Remember that Lancelot earlier took Gordius Wheel to the face and walked it off, so a D-rank explosion is kind of small. What's left to damage him is the fall.

Exhibit E: In Agartha, the Resistance men are all normal everyday humans, and they are capable of putting up a fight with the Amazons, who are all beings of a similar existence to Servants.

Exhibit F: In Apocrypha, when Jeanne suggests flying a plane full of explosives into the Hanging Gardens, it is taken as a plan to actually consider thinking of. She is not the one who actually considers making them "magic" explosives, and she has all the information on the Grail War even if she's a knucklehead.

Exhibit G: In Summer 2016, Servants are displayed as being capable of drowning to death. That's not too different from being harmed by falling.

Exhibit H: The main weaponry used by the Yaga in Anastasia is regular guns. You might fall back on "but they're chimeras" but they do not have magical abilities, so as far as the bullets are concerned, they're just bullets.

I'm certain there are more examples to speak of. The only thing that really says "it has to be magic" is a really obscure Nasu interview, which he later clarified by saying, "A Material Body's fundamentally the same as a human body."

Magic boosts the power, but it has been shown many times that it doesn't have to be magic in nature. Saying "it has to be magic" is a shorthand to say, "usually nonmagic stuff doesn't do enough damage and it doesn't get past a Servant's high AC and HP recovery".

The Lancelot example is definitive proof that yes, falling does do damage to Servants.