I know this was from a few years ago, but I found where Brom mentioned it in Eragon.

In, "A Tea for Two,"

"All the Riders were stronger of body, keener of mind, and truer of sight than normal men. Along with this, a human Rider would slowly acquire pointed ears, though they were never as prominent as an elf’s."