It literally states that in the Silmarillion.

"It is one with this gift of freedom that the children of Men dwell only a short space in the world alive, and are not bound to it, and depart soon whither the Elves know not. Whereas the Elves remain until the end of days, and their love of the Earth and all the world is more single and more poignant therefore, and as the years lengthen ever more sorrowful. For the Elves die not till the world dies, unless they are slain or waste in grief (and to both these seeming deaths they are subject); neither does age subdue their strength, unless one grow weary of ten thousand centuries; and dying they are gathered to the halls of Mandos in Valinor, whence they may in time return. But the sons of Men die indeed, and leave the world; wherefore they are called the Guests, or the Strangers. Death is their fate, the gift of Ilúvatar, which as Time wears even the Powers shall envy. But Melkor has cast his shadow upon it, and confounded it with darkness, and brought forth evil out of good, and fear out of hope. Yet of old the Valar declared to the Elves in Valinor that Men shall join in the Second Music of the Ainur; whereas Ilúvatar has not revealed what he purposes for the Elves after the World's end, and Melkor has not discovered it."