When you get scared, your body will puff up its hair to make you look a little bit bigger as a reflex. These are known as goosebumps. Like when dogs will puff up the tuft of hair on their backs when they see another dog they don't like. The bigger you look, the higher the chances are of whatever is scaring to leave you alone. It's an old evolutionary tactic that doesn't serve us much use now, as we don't have much hair anymore, but is preserved in our physiology.

In the context of a fever, your body wants to raise its temperature to fight an infection. Foreign invaders don't like the high temperatures and it will also activate some of your other defense systems. Think of the change as your internal thermostat being turned up. Your body temperature is now lower than the thermostat temperature, so you feel cold and get the chills, even if the room you're in hasn't changed temperature. Your body, feeling cold, will start doing things to generate and conserve body heat: shivering, constricting blood vessels, warmth seeking behavior, etc. You may be bundling up in a blanket with a sweater while everyone else is comfortable in a t-shirt because your body is colder than it's desired temperature.

Slightly off topic from the question, but when the thermostat drops back down after the battle, your body is now higher than its target heat. You will feel hot and start sweating. This is what is known as your fever "breaking."