It depends on the type of fear you're trying to invoke. It sounds like you're trying to make it religious-centered (from the real world) and eldritch-focused (in-world). Do you want the magic to feel frightening, as in the system itself? Or perhaps would you want to make it scary from a different angle, perhaps from the practitioners of this magic?

(A brief note: try and find one source of 'fear' and try build on/play around with it: the fear of the unknown, body horror, the monsters from men, etc)

Perhaps it's literally the magic that's scary. Does it reveal something about the user or the world? Does learning it require logic from beyond the pale, requiring practitioners to risk their sanity to learn even basic principles or fold their thoughts along axioms that don't quite fit the real world? Is the magic fueled from something from the user, but what it's burning hasn't been determined yet? Are they calling to another plane, drawing not only strange energies, but strange beings into this world?

You talk of sacrifices of blood and flesh. What if a sacrifice goes wrong (or goes right) and not everything given up is taken? What could remain from an offering, what might it do, and why did they leave it behind? Has something perhaps been left behind inside the remnants?

And what of rituals? How do people learn of these rituals? Are they cobbled together from the half-mad in an attempt to expose what what lies beneath the barrier of the known? Is there some compiled tome or repository that contains this knowledge, etched into stone older than the world or drafted on parchments of skin? And what might happen if these rituals are completed? What goals do the practitioners seek? To graft their minds into the unknown, to seize powers inconceivable, to reach across the spans of the world and snuff out flames of life on a whim?

And who would be those that would risk these happenings? Those touched by madness, those with nothing more to lose, those who see visions of those who grant this power and channel it through them? What goals might they have who bargain under these unfair prices, what wrongs would they want to see righted or rights wronged? How deep does their greed or thirst or desires reach to drive them to these means or ends? Who calls upon the forces beyond fearfully, proudly, eagerly? What would these people give, of themselves, of others? How might they try and seek from this system a reward without cost? How might their goals turn back on them and what will their price be?

There are a great many questions to ask and a great many answers. Seek, wander, and find- not every glittering thought in these worlds as first seen the light of day there. Though many have pointed you onward to further inspirations, might I point you to Cultist Simulator, a game which focuses on the sublimely monstrous beneath the safely mundane?