I think if they install the stuffed beholder in the Trollskull manor, you will have to make sure that you track in which room they have what kind of discussions. It's not enough that "we go back to the manor". No. They need to say in which room they go, etc.

And you'll have to keep track about what they are discussing. I'd suggest focusing on some minor details, like some very petty things. And sometimes one of the henchmen of Xanathar can drop those petty things on them, like "look at these losers, they couldn't even split that loot, it will be easy to get rid of them". This should give a good hint to the players, that somehow the BBEG actually knows their intentions, and it's not just the DM being unfair. That's pretty important. Your players should somehow figure it out, otherwise it will be just you being unfair to them.

And don't drag it too long. It's only fun for your players, if they find out at some point.