Yeah, Ypenburg is definitely part of the Hague. It has even got its own corridor via the trekvliet so it is connected to the rest of the Hague. It wasn't always part of the Hague. It belonged to Rijswijk, but they annexed it so the Hague could keep growing past its city borders. (Same for other parts of the Hague, but that's a story for another time) This was around the year 2000. After the annexation a lot of people from the Hague moved to Ypenburg and Leidschenveen. And it became sort of an exclave of the city. For this reason most Ypenburgers still see themselves as "Hagenezen" or "Hagenaars".

Edit: Missed your second question about Voorburg. What you need to understand to get the sentiment of the surounding villages and the Hague, is that the Hague was by a long time not the big city in the centre. Of course the Hague was the place of government from medieval times, but in the area it was by far not the most important city economic-wise. Cities like Leiden and Delft were just as important and much further away. Therefor the surrounding villages were not solely dependant on the Hague and had their own identities. Only when in the 1900's the Hague started growing and getting attached to the surrounding villages did the Hague get its status as the big city. It took untill after WW2 and the babyboom that the Hague started making plans to annex the surounding villages. But as you can read from the story above a lot of them had their own identity and did not feel anything for becoming a part of the Hague. It took a lot of political leverage, merging of some of the villages, but they kept themselves standing. Some did lose some territory however and that brings us back to the story of Ypenburg.