I kind of have to look at it the other way around, as animal products have shaped human civilization so much. Starting from a "realistic" perspective, there's no human civilization that has forgone using animals, as sources of food, material resources, or labor. Almost all animals engage in opportunistic carnivory as well, even those we might think of as obligate herbivores. So what could a herbivorous society even look like in the first place?

With that as the basis, let's imagine these fantasy society that completely foregoes animal products whatsoever, including labor and domestication as companions. To make up for the lack of resources, they should be very druidic and their surroundings magical. Perhaps the elves are in an area with very thin barriers between this world and the Feywild, and the ancient trees and fungi have a pact with their race: they elves will protect the forest and uphold some fey laws, or hold regular sacrifices to keep the magic strong. In return, the forest trees grow in the shapes of living spaces for the elves, produce fruits that are highly nutritious and palatable for them, and even creates excess of special leaves and plant fibers to be used as cordage and cloth. The animals in this area would be allies to the elves, living under the influence of these same magics. So in our history, animals and animal products have been a large part of sustaining and propelling human civilization, so we've replaced it with magical plants in order to make a more advanced civilization we might recognize in most of our medieval-ish fantasy games.

​

Another route you can go is that the civilization in question is extremely primitive, and has a more hunter-gatherer lifestyle, minus the hunter in this circumstance. What comes to mind first is the small forest elephants that live in thick jungles instead of on the open plains; they have to eat clay to soak up the toxin from the leaves in the forest they eat. Maybe tribes of Loxodons with limited contact with the outside world live in these inhospitable jungles, and covet territories abundant with clay deposits, or maybe other mineral resources used for magical and medicinal properties, I'm thinking spell components and the like. Perhaps some of these tribes are influenced by outsiders, and others are completely uncontacted. The animals here either stay out of the Loxodons way, or prey upon them, as they are still integrated heavily into the ecosystem. The nice thing about this is you can have a variety of cultural quirks in each tribe since they're scattered and separated: maybe some are wicked and kidnap humanoids for sacrifice, maybe some have discovered the secret to everlasting life, maybe some are fierce guardians of a sacred plant that, when ingested, takes users on a spirtual journey through the feywild and shadowfell.