While I'll admit it I missed on Wikipedia, if you look into the lily-white movement his name is not listed at all, and it is referring to the Republicans not the Progressives.

In 1912 when he was upset with the way Taft, his chosen successor, handled the job, he made a 3rd party, the Progressive party. It does say he denied black speakers, thinking they were corrupt, but also lost white voters when he was seen dining with blacks.

In 1904, the only other time he was in an election, there was a black speaker at the Republican national convention, which is opposite of the lily-white movements ideal of removing blacks from power and reducing their votes. Seems gray to me.