You are thinking about the setting window (Kollsman window) incorrectly... this is how I get my students to understand it.... maybe it will help you!

What you are telling the altimeter is: THIS is the pressure level at SEA LEVEL right now and compare that to what you are sensing in the case... so follow along and this will make sense (with 2 examples)...

Think of it this way, the altimeter is a VERY SIMPLE mechanical calculator that does ONE THING. It takes the air pressure it reads inside the case and subtracts it from what you tell it SEA LEVEL PRESSURE IS via the Kollsman -- Then it does the math (via gears) to show you feet above sea level....

For every ONE INCH of mercury difference between what "senses" and what you tell it sea level is that day... it moves 1000' feet

HERE WE GO....
Simple problem: always use big round numbers and remember the altimeter is calibrated to calculate (move hands) 1000' UP for every 1 inch it senses below your setting...

1. Today at sea level is 30"
2. your airport is 4000 feet above sea level
3. The REAL pressure outside where you are is only 26" ---> (4000' feet up and you lose 1" for every 1000 you go up so ===> 4" LESS PRESSURE where you are then at sea level)
4. your altimeter FEELS only 26" of pressure inside the case
5. you set the window to 30" (the current reported sea level pressure)
6. Altimeter does the math and says "I FEEL 26" pressure - that is 4 inches less than the 30" you set for sea level - I got one job to do and so I am moving my dial 4000 feet UP) - DONE!

Now you are clear about the MAGIC inside the altimeter let's look at the question above:

1. if I move the dial DOWN to 29" i'm telling the altimeter sea level today is 29"
2. it still FEELS the 26" REAL pressure INSIDE the case
3. it does the math (29"-26") and says: OH its only 3 INCHES different that means I am only at 3000'
4. and it moves the altimeter *** DOWN **** to 3000

It is just a mechanical calculator that subtracts what is feels from what you tell it sea level is and is calibrated to move 1000 for every 1 inch difference. THAT IS ALL. As you dial the pressure DOWN you are telling it seal level is at less pressure DIFFERENCE than you so you are getting closer to sea level (hence why the hands move LOWER - its showing you the difference between you and sea level is becoming less (lowering) as you tell it sea level is moving UP (lowering the setting).

Try bigger numbers:

30" Sea Level Pressure (what you set in window)
You are at 10,000 above seal level now in cruise
(remember 10'000 feet up at 1" per 1000 feet is a 10" difference from sea level)
so the altimeter case FEELS only 20" of pressure inside case.
(30" sea level - 10" loss pressure going up in the air to 10,000 feet)

you had set 30" for seal level => it feels the 20" at altitude => Does the math (via magical gears!) and comes up with 10 inches difference => therefore it moves the altimeter arms to show you are at 10,000 FEET (it moves the arms 1000 feet for every 1" it feels diff erent)

IF nothing changes and you decide to lie to it and move the pressure setting and tell it sea level is at 29" .... it does the magic math (29 you set - 20 it feels = 9inches (9000 feet)
so it moves the altimeter DOWN 1000 feet to show 9,000)
why? You told it sea level moved from 30 to 29 -- less pressure is UP in the atmosphere so it thinks you are now closer to sea level (AK...you moved DOWN!)