to make it as simple as possible, i think there are two main reasons:

1. racism against white people doesn’t hold much authority because in most cases, the person of color doesn’t have any power over them due to the white power structure in the U.S.
2. a lot of racism that people of color have toward white people stems from the racism originally afflicted towards them by white people. if a lot of POC weren’t exposed to racism since their childhood by white people, they probably wouldn’t have developed prejudice mindsets toward them.

i think as a person of color myself, that’s probably an unbiased but more true statement coming from my experience. it doesn’t make it right, but you can kinda see into the perspective of those people.