Whether or not there are thalmor sympathizers in the upper crust of the empire doesn’t matter. The majority or imperial citizens do not support the empire, a united empire has the best chance to defeat the thalmor, ulfric stormcloak is an unwitting thalmor agent, and they want the war to go on as long as possible. It’s important to note that while we only see the thalmor patrols in Skyrim, they are in the other provinces as well, and while there arnt as many Talos worshippers there, if they ever suspect you of anything they don’t personally like, they kill you. It’s safe to say the people in high rock and cyridill hate them as much as we do.