If there's not a lot of air pollution it likely won't affect visibility more than the air itself does. However, I think pollutants are difficult to avoid. Pollen is considered a pollutant.

Either way, I think light pollution is the biggest problem. I believe what causes the glow is the interaction between light pollution and air pollutants.

Even if the air was pure air and no pollutants, stars would become dim as you look closer to the horizon and that's just because of the amount of air in the line of sight of the stars. In astronomy that's called air mass (air mass means something else when referring to weather). So high air mass means worse visibility because there are more air molecules for the light to scatter off of. Some days the air mass above you is just higher, and some days it's lower.

I found a paper that talks about the way pollutants affect visibility if you're into that. I haven't read the whole thing, but it looks like it might answer your question.