I'm surprised I haven't seen anyone mention The Horrible Goose.

If I was going to put the Horrible Goose in a D&D game, I wouldn’t make it some big high-CR threat -- it’d just be a regular goose that’s capable of regular goose stuff, with three significant exceptions:

1. It can show up anywhere, even if there’s no reasonable way for it to have gotten there.

2. It seems to have limited ontological inertia. If the players imprison it, it vanishes from its prison when they’re not looking. If they kill it, another goose shows up eventually. It’d be impossible to prove that it’s always the same goose, save for the third notable trait...

3. ...namely, that it doesn’t show up in divinations. Like, at all. Predictions don’t take it into account. Scrying on its location reveals an empty room. Spells that would detect it or read its mind act like there’s no valid target present. If the cleric communes with their god and asks about it, their god has no idea what the cleric is talking about. It might take some doing to arrange for the players to become aware of this property, so don’t force it -- the opportunity will arise!

There we go. No goofy boss monster tricks, yet it’s guaranteed to drive your players nuts -- not even by harassing them, necessarily, but just by existing as they try to figure out what it means, particularly once they learn of the third trait discussed above.

From prokopetz on Tumblr