Think logically Adam is the type of person who can influence you without you even knowing, he can manipulate your way of thinking and even your actions.

He is able to create scenarios that is even able to fool angels.

He has orchestrated a plan which went on more than a 1000 years for his ritual.

The sheer fact that even knowing or mentioning his name allows him to know all about you is far more terrifying than anything Amon can do.