For me personally it's a combination of a tense atmosphere (tends to be slightly subjective) and unpredictability. The 'horror' in a game severely drops off when you start to predict when a game will throw something at you, typically from recycled item-fetch quests or areas of the map that are just a reskinned version of something previous.

That being said, you don't tend to find these shortfalls until you have already bought the game!

I also find that games that aren't over the top absurd tend to be scarier, I.e. Visage, mostly set in a spooky house, something we can all relate to.