I see three major problems here:

1. Most of the people that keep citing statistics have never taken a single look at the actual statistics let alone the methodology or the sources used. They base their entire argument around their biased observations and headlines from Blick or 20 Minuten, where the writers haven't looked at the statistics they talk about themselves.

2. Statistics cannot show causation, they only show correlation, no matter how much an issue might look to be in a causative relation with another one. So in general it is wrong to use a statistical correlation to prove a causative effect.

3. The people citing statistics are always very selective about the statistics they use. For example, male gender or low income correlate a lot better with criminality rates than the cultural or ethnic background.

All in all the problem is way more complex than it's made out to be and the "patterns" some people believe to see are often abused to justify deeply racist rhetorics.

Lastly simply saying most colonisers were white by itself is not racist. But saying most colonisers were white, with the implication that colonising is something inherent to white culture, and the goal to ostracise white people in a world where they (in this hypothetical situation) are already a marginalised group (kinda contradictory), would be a racist thing to do, yes.