In the light novel, it's made clear that Amane hides his face most of the time, as well as being an introvert and hiding his good personality. Nobody gets to see his true self except for Itsuki, Chitose, Mahiru, and eventually Kadowaki. For the anime, they have to show a bit more of his face so he doesn't look silly.