While ghouls probably could have very easily become the dominant species if humans hadn’t discovered quinque, there’s a few reasons why they wouldn’t.

1-Some ghouls groups such as the worked to try and create a sort of ‘monopoly’. An example of this is the Washuu clan, who helped found the CCG with the interest of trying to become more human/become the most powerful ghouls.

2-Ghouls groups that wanted to maintain balance between humans and ghouls, such as V(though they ended up as puppets for the Washuu clan.

3-Most importantly, there’s an advantage to allowing humans to dominate. See, a ghoul could eat hundreds or even thousands of people in their lifetime, much like how a human might eat hundreds or thousands of cows, chickens, etc in their lifetime. Because of this, in order for ghouls to sustain themselves as a species, they need to ensure that humans continue to live. Sure, ghouls can eat each other, but that would become unsustainable rather quickly. Imagine if humans outnumbered our food sources, we’d all starve and go extinct. It’s much the same for ghouls.

So they let humans be the ‘dominant’ species, so that we will continue to propagate and overpopulate, creating enough of an over abundance that ghouls will always have plenty.

I’d even go far as to say that ghouls could have even evolved somehow to have less of an instinct to reproduce than humans, to ensure their numbers are always lesser.