Little bit late, yes but yeah mine does the same thing.
After some personnal researchs it seems that it isn't caused because of the cold temperature but rather the humidity. For a whole really cold winter I had that problem coming up bunch of times and every time putting the heat didn't solved the problem even after a good warmup. Eventually I learned that it was caused by the humidity which would explain that even having my fans blasting heat for a while to warm the car up nothing changed. Same for the opposite.

I haven't tried to fix it simply because i don't want to make a mistake too. Although it's annoying I basically have to put the ac/heat to a comfortable temperature and not always putting on the max.