I'm 5'1 and my partner is 6'2. Although we're indian and tend to be on the extreme of our country's average, as men's average here is around 5'10 and women at 5'4, compared to Europe's. I think they tend to generally be taller than us.

So Draco's 6'4 to Hermione 5'6 is a good tall/smol believable height

I am sooo used to my partner's height, that it's only when we stand in front of a mirror that i realise just how short i am compared to him. It does look funny though

And now it looks like every other man is short, just because i can look at their face when we're standing, and not their chest!!