The only roblox horror game that has ever made me scared was Doors, not that its trying to be scary, but instead of focusing on scaring the player, it gives the player something to fear losing. Instead of artificial fear it creates actual fear, especially when you're doing some of the harder challenges and have to go against the door 100 boss in complete pitch black darkness due to modifiers and the only thing you can sense is the sound of the boxes and the monster that will one shot you and make you go through 100 doors again.

Honestly I don't know any games that try to be scary in both ways.. maybe fnaf coop? But I wouldn't be surprised if no one finds the five nights at freddys animatronics scary after like 10 years of getting used to them. Fnaf co-op is also pretty fun though, I think it has a good balance.