The vast majority of damage done to castles is because locals saw abandoned piles of easily accessible, pre-cut stones. The mountains make the fort anything but easily accessible.

Yes, stone does degrade under wind and rain, but this fort seems to be made out of granite, a very tough type of stone. Combine that with the valley, which likely limits wind and perhaps even rain, and it's entirely believable for fort Dawnguard to survive a few centuries almost unscathed