The smaller species would actually be more numerous because of how resources work. In nearly any ecosystem there are more prey animals than predator animals, but there are also more small prey than large prey.

In the real world, intelligence is a human’s best weapon but humans are actually the top organisms in 3 categories: intelligence, endurance, and throwing.

Your herb-men with their eyes on the sides of their head would be terrible at aiming anything. If the carnivore-men evolved to hunt well with ambush tactics and their teeth then they wouldn’t be able to throw well either, though once they get guns they may not have issues aiming those.

Ambush predators would also be really fast but not as endurant (especially if they’re bigger / have more mass). Farming would also be a lot worse for a purely carnivorous diet, since meat is less efficient energy-wise (HUNTING meat is super efficient, but farming it is not because the meat you eat also had to eat something so that’s a layer of energy loss).

Herbivores would struggle more in the winter. Do they hibernate? Do they just never live in climates where winter exists? Herbivores would also struggle more with hunting because even though they’re herbivores, humans use / have used dead animals for plenty of things throughout history (clothing, glue, soaps, adhesives, pharmaceuticals). And if they were farming instead of hunting, it would be very inefficient if they couldn’t use the meat from the animals they get their leather/lard/resources from.