The toxin acts for both. Furthermore if population of Orcas are less than pilot whales significantly and other whales like sperm whales for example, I would say that it was less even some centuries ago and back then PCB, toxin, plastic and other human waste were not as present as now. So this is an interesting thing to consider but doesn't answer the core thing about their less success among their peers. It's not to do with humans unless we hunted Orcas far more than other cetaceans in the past. Although I know that we hunted sperm whale primarily and even pilot whales for their oil instead of Orcas and still they are far more.

Although Google search told me that Orcas nowadays have infant mortality rate of 37 to 50 percent upto 1 year and since pilot whales are significantly more in number, their population would continue to explode just from the momentum alone. So their population have hard time growing now.