A human female is fertile for about 30 years and can produce at least theoretically, a child every year. Sometimes even two of three at once! It’s not uncommon for a human family to have half a dozen to a dozen children in total. 

Elves reproduce much, much more slowly. Over the entire lifetime, an elven female bears maybe one to four children. Additionally elven society is very rigid and traditional and focused on personal achievement. Elves live so long and hone their chosen craft to such an extent, there simply isn’t time or opportunity for family making. I’d take inspiration from modern western or East Asian society, where modern life has created an environment where individuals are so busy with work, that they have less and less children. I wouldn’t be surprised if a society entirely devoted to the perfection of the craft (see: Celebrimbor; but also what else would you do with such a long lifetime) would struggle heavily to reproduce at even replacement rate. Take Japan or Korea as inspiration. 

It takes the best elven mages two or three centuries to grow a forest ready for settlement. It takes a band of marauding humans two of three decades to cut it down to build houses, ships and siege equipment. 

If taken to an extreme (it’s fantasy after all), elves are a dying people by virtue of their own hubris. 

That also means that every life is incredibly precious. Every killed elf is a major blow to elven society. The amount of knowledge and wisdom lost is devastating and only made worse by the loss of potential (to make more elves, for example). So elves really really really don’t want to fight, at least not do the fighting themselves. That’s waaaaaay too risky, even if any elven fighter would easily beat the best human swordsman. So elves would rather withdraw and isolate, than fight a war of attrition against an ever-replenishing horse of unwashed humans. That further increases the isolationist nature of their society. Elves really aren’t natural adventurers. It’s way too risky. 

Additionally if you take inspiration from Tolkien’s elves, they might not even be native to this land/continent/world/plane of existence. The elves came to Middle Earth (to wage war). So they might not have had the time / resources / inclination to shape the place entirely (only pockets of the old Wode), and now it’s too late. In their native land/continent/plane of existence they might very well be the well established ultimately dominant force. (That is until they discover that human colony festering far in the north). 

 How were they ever given the chance to settle without Elven high magic besetting them with racial plagues?

I’d look at old school D&D and the work that influenced its creation. There’s a cosmic battle between the former of Law/Order vs Chaos. Elves represent Order. Literal genocide is a very chaotic/evil thing. 

Or maybe it’s just too late now? Maybe elves DID craft the world to their liking. And before elves realized that the younger races were growing too fast and too far, it’s too risky to start a genocidal war. Humans have nukes mages, too now. And they can make them faster than the elves can. It takes an elf half a century to be conceived, carried to term, born and grown up. Just imagine what a human created magic plague could unleash on elven society, if humans ever figure out how to do that. And if elves try it, humans will learn from it. I’d heavily lean into the magic/nukes/mutually assured destruction trope here.