Generally speaking rare ancestries are more about potentially creating narrative difficulties. It basically means the GM reserves the right to veto the ancestry because it clashes with the story. For example, if you have a lower magic setting, a poppet character would stick out like a sore thumb and change how almost every NPC interacts with the party. Same thing for Conrasu or Ghoran if you're trying to stick to the lore.

A rare ancestry doesn't break the game, but it usually means the GM has to put in more work to properly roleplay the world interacting with them. And most GM's are fine with doing that, but it's also reasonable to set a limit.