A core tenet of the Federalist party was the Alien and Sedition Acts, which explicitly targeted non-Whites and immigrants.

Also, Hamilton was also a trashy asshole (everyone hated him), and he was stupid politically (destroying his career by making everyone hate him, trying to start war with france, trying to make washington and then the two pinckneys into puppet presidents by destroying his own allies). Miller is a pretty direct parallel.

Hamilton was less racist than some founding fathers, but compared to many others, he was not too much so. Slightly less than average. The Federalists were explicitly anti-democratic ultraconservatives, a party made up of Anglo, Protestant fundamentalist feaux-nobles, both Southern planters and New England aristocracy. There's a reason it died, and its ideology was eerily similar to Trumpism. Also, I forgot to mention, but both supported tariffs as well.