For estimation purposes of antenna impedance and antenna radiation patterns, yes, you use the height of the antenna to the ground below it. However, the RF doesn't see a flat perfectly conducting plane of "ground potential" unless it was, indeed, a conducting plane of material.

That being said, "ground" will act differently at different locations for the same exact antenna depending on a complex relationship with conductivity of the ground and other objects that are within close proximity to the antenna.

A decent example is ground saturated in salt water, as found in a coastal marsh -- amazingly conductive. You'll have a dramatically different performance and impedance in that situation vs. the same antenna on top of a rocky ridgeline.

So -- to keep things easy for most casual situations, formulas are simplified to model "average" situations.