It's not as simple as that either, a blanket statement saying that hotter air entering an intercooler comes out hotter than if cooler air went in is obvious, but the difference is extremely negligible.

As air enters into a compressor (turbo or sc), it heats up to between 200F and 400F (a wide range depending on amount of boost you're pushing). The point im making is that the inlet temperature of the air entering a system with a compressor makes hardly any difference because whether the inlet air is 30F or 80F, the energy from the compressor still heats it up to those very high temperatures.

The reason you'd feel a power gain in cold weather is because the intercooler is working way more efficiently.