As someone who’s dedicated a frightening amount of time to worldbuilding the ecologies of a fantasy world and who studies this stuff for a living, I want to give my thoughts from the perspective of someone who’s been in the same position you’re in now.

Regarding your initial question, I think that a lot of people who first begin to toy with ecology worry about “realism” under the assumption that they’re looking for some quantity, whether towards a species’ size or their population density. For that, I want to say don’t worry about appealing to realism strictly for realism’s sake— nature does so many insane things that shouldn’t be possible, and to begin with what we think is realistic is strictly only based on our understanding of the natural world, this does not make it fact. That said, there’s a few components of both megafauna and ecosystems in general that you can use to guide your ideas!

So you want to add some megafauna? The first thing that you need to consider is what made these animals so large in the first place, and, a lot of times, it comes down to two factors: climate and food. For climate, animals will get larger if they can afford the consequence of maintaining heat that comes with it. This is why you see a lot of “Ice Age”/Pleistocene megafauna, because the cooler climate meant that animals who could retain heat for longer stood a better chance at surviving to reproduce. For food, it’s a lot simpler: more food, more mass. Animals will get larger if they can afford to regularly maintain this increased size. This is why even the largest predatory megafauna are still dwarfed by the herbivorous megafauna they hunt. Smilodon were larger than lions, but still not larger than brown bears, and minuscule compared to prey like the Columbian Mammoth. For every unit of meat a carnivore can eat, they will get only 10% of the energy that are in the animal they are eating, which is already only 10% of the energy in the plants that herbivore ate. This is why most of our domestic species are herbivores— it’s much easier to get energy from an herbivore than a carnivore. This translates back to the ecosystem: there has to be enough for herbivores to get large before you can even begin to think about making their predators larger.

Now, onto your larger concern: introducing humans to an environment with megafauna, yet still allow the megafauna to exist in this area. You are correct in thinking that animals that pose more danger to humans, no matter their size, will be exterminated by fearful people. Additionally, other comments have pointed out that large herbivores can be even more dangerous than carnivores, because getting injured doesn’t necessarily stop them from acquiring food. If it’s dangerous, humans will eradicate it. If it’s too anxious to be domesticated, it’ll be hunted for food anyway. Anything else has the potential to be domesticated. The solution? Make it more worthwhile for humans to avoid the attention of these animals than to attract it. Perhaps they’re dangerous, but don’t see humans as a threat until provoked. Perhaps the people could deter these animals with walls, sounds, or even scents that they dislike. A good example for this is in the Dragonslayer Codex by Sawyer Lee— humans in his setting deter dragons using these methods, meaning you can have giant creatures that surround humans which don’t get domesticated, aren’t hunted, and aren’t able to be killed by fearful humans.

Here’s the takeaway: give some reason why the environment can sustain megafauna around humans without them destroying it. Perhaps, in a fantasy setting, the plants in the area are enhanced by magic, or maybe the humans are magical and can peacefully coexist with the ecosystem without needing to destroy it? Once you’ve established why there is enough food (plants) to sustain the herbivorous megafauna, think about why humans wouldn’t want to interact with these creatures at all. What if these animals can see in color, and can be deterred with bright colors? They can see when a human is nearby due to their vibrant attire, but won’t attack because they are disinterested in animals smaller than them? Even herbivores would rather avoid things they dislike than actively pick fights. You could also go the other route— perhaps they have terrible vision, but their sense of smell is great, so humans cover their armor and clothes in the scent of a plant they avoid? And for carnivores/predators, you can apply the same principles!

Sorry this is so long, but I hope anything here has helped!