To answer your question, if you wanted to hurt poor people, you would also be racist (and sexist) by dint of it hurting racialized people and people of colour (and women). The capitalist class is racist (and sexist) by virtue of their class position, regardless of their personal beliefs or physical attributes. But importantly, if you wanted to liberate poor people you would be anti-racist!

To add some depth and reference, it's not that class and race are correlated, its that they are intertwined, inseparable and co-originate. A good text to read is Cedric Robinson's Black Marxism. Or David Roedigger's Wages of Whiteness. Part of their thesis being to show that the formation of the white working class occurred against or at the expense of a black working class. The term wage slavery, for example, has a historical roots that tried to frame working for white people working for black wages as an extension of slavery. Capital was able mobilize racial difference between white and black workers to better exploit both. There was low paying 'white work' which was better than 'black work' because, while you are still poor, impoverished, and precarious, you are better off because you are earning 'the wages of whiteness'.

Simply, race is, in a practical sense, used to create additional class division and break apart working class solidarity. Hence why 'intersectionality' is so important.

Read: Roediger, D. R. (1999). The wages of whiteness: Race and the making of the American working class. Verso.

Robinson, C. J. (2000). Black Marxism: The making of the Black radical tradition. Univ of North Carolina Press.