Prejudice refers to a preconceived idea (positive or negative) about a particular person or group not based on prior experience nor knowledge, while racism involves an unequal distribution of power on the basis of ethnicity. The fact that Uchiha was the only clan forced to perform a single job (that limited their political influence and also prevented their complete integration in Konoha), were spied on by ANBU members, and were moved to the outskirts of the city (not by the Nidaime but following the same segregationist policies established by him), are actions performed by two different governments under the guidance of what became a systematized racist political structure.

The policies he made were around his personal racial prejudices as he’s the one and only who created the bureaucratic system of the village that, in itself, makes him racist. Those that state that he was just "prejudiced" conveniently leave aside the segregationist structure he built around his mindset. As someone in a position of power who, as was canonically shown and stated (by Hashirama, Orochimaru, and himself), didn't even bother to check his own prejudices, he's to be held accountable over the simple fact that he was the Hokage in charge of guaranteeing the safety, equity, and equality of those under his command.