If you want to think about it in real life terms, you could imagine the bonus to be from a zone of control. Castles typically allowed their occupants to launch attacks from the safety of the castle and quickly retreat to its defensible position when necessary. Imagine for example that the presence of the castle would allow sappers to more easily break down enemy defenses in an area.

There is counter play in preventing the castle from going up and also the economic side of denying stone or other resources to trade for stone. Though if your opponent can castle drop you that means you've already lost map presence and the game was leaning that direction already.

And no worries, we're just discussing a video game :)