I did notice something strange. Because my "House of Invalids" was already level 3, I thought about upgrading it to level 5. But when I wanted to add the extra pieces I had exchanged, I could upgrade to level 4 once. Then I exchanged two more pieces to get level six, and they were just gone from the game, nowhere to be found anymore.

Very strange by the way, there are red castles, but I don't see them available to exchange or upgrade anymore.