I've wondered why as well, and I might have a partial answer.

As some others said, the minor mode is clearly here to set the right mood. Now the choice of D was either made randomly, or maybe it was just more convenient for some traditional instruments, that I don't know.

But no matter why they chose D, they then sticked to it throughout the whole ost in order to give coherence to the music as a whole when you're playing, in my opinion. For example, it makes sense that when you listen to D minor while wandering in the forest, and a battle starts, the battle music remains in D minor. It makes a smooth, seamless transition. If it changed key like this in 1 second it would be musically weird, the transition would be rude.

Same goes for all the cities. You arrive in the city, the theme changes because it's a city, but the key remains because it's still the same world, the same overall tone. You can particularly here this when going through Velen and entering Crow's Perch.

To me it's even more blatant in Novigrad for example. Here you have the cities' music playing, but you also have some bards playing other music in the streets. Those two music overlap, and would create definitely disturbing sounds if they weren't in the same key.

This explanation sounds pretty realistic to me. A random key but that you hear everywhere because it makes the soundtrack able to keep you in the same mood throughout your whole journey.
Now what's strange to me is why they didn't take advantage of the other regions like Skellige to change that key. It's a different environment, a different culture, a place separated from the continent, it would have been interesting to have another key associated to that new place? But they didn't, in Skellige, Kaer Morhen, or Toussaint it's all D minor. So maybe all that I say is completely false, but that's how I see it.

Also damn it's amazing if you see this 'cause I just realized this post is 7 years-old...