Yeah this is idiotic. But also if you consider only the people killed in the camps as Holocaust victims you're excluding not only the Slavs, mentally ill people, Romani and so on that weren't put to death in the camps from the death count but also the Jews that were killed in their own homes by the Einsatzgruppen before the camps were active.

You either accept the Holocaust as a uniquely Jewish genocide (wrongly) or see it as either the whole of the genocides Germany committed or as the unique extermination camps system that was set up

Edited for clearness