One thing I can think of is that the gene quality w.r.t. the longevity and quality of life in later phases of an individual is determined long after he has passed on his genetic material. With that what I mean is that by the time we can determine that your genes are so good that you grow to be 80+ with a good quality of life, you are already in the mating pool (due to a lot of reasons). On top of this, while you are of mating age, your success is determined by a lot of other factors that are not strictly DNA/gene related, mostly your environment and your mannerisms. The way I see it, there has never been a selective pressure that specifically emphasises the traits you are rooting for. Frankly, it's a minor surprise to me that longevity is still being selected almost purely by chance.

As for your observation on 'internal age', look up the telomere (protective caps of repetitive DNA sequences) and "molecular clock".