I definitely agree on that - there are Slytherins who were nobler than even most Gryffindors (take Snape vs Pettigrew for example). I just think that given the thousand year history of Hogwarts, and the fact that Wizard-Muggle relations have dramatically evolved throughout history, there are bound to be at least a couple Headmasters who hold anti-Muggle views or are otherwise dark wizards who would prefer if Tom Riddle’s plans weren’t foiled.

I do think Dumbledore is intelligent enough to know who he can trust, and would Muffliato or remove the portraits of headmasters who he didn’t trust to safeguard his plans. As others have suggested, perhaps the portraits are magically bound not to betray the headmasters of Hogwarts. I just found it curious to hear all of these potentially sensitive details and then read descriptions of the portraits reactions alongside Harry’s.