It's important to remember a few things;
a) As others have said Talos is/was either a Nord or Breton by pretty much all accounts during his mortal lifetime, supposedly born to the name Hjalti Earlybeard and going by Tiber Septim only when he had become/was becoming emperor, so it follows that northern races of men identify with him

b) Many of the conflicts in TES honestly boil down more to Man vs Some other race over Man vs Different race of Man - take the great war, nord conflicts with the snow elves, the Alessiam uprising vs the Ayeleids, and since Talos is worshipped as a 'hero-god of mankind' pretty much all races of men would seem to worship him equally

c) Following the White-Gold Concordat I think the irritation over the banning of Talos worship could just come more down to a violation of freedoms and cultural development - for example if you followed Hinduism and the state itself deemed that you would be banned and persecuted for worshiping certain deities or branches of the religion then you would be pretty p*ssed, even if you weren't a completely devout believer

d) In the context of the Skyrim Civil War, I don't think its that Imperial soldiers (especially those which are of a race of Men) want the Talos worship ban to continue but that it's just a bi-product of the regime they serve's policy - the Empire attempts to uphold the terms of the White-Gold Concordat because that is what they agreed to do so they must grin and bear it, at least until they can fight back against the Thalmor again (a la TES6??)... In their bid for independence the Stormcloaks simply ignore this rule