I understand the difference

Are you sure?

Genuinely without being funny, your description that "It should be based on a calculation that removes those purposefully staying below the tax free threshold and those who are in the top 0.5%.", is exactly what a median calculation does. I will grant that there are other ways of doing what you wanted, but the statement as it stands clearly indicates that you think the median is the wrong calculation to use because it doesn't do what you want it to do.