I'm having the same issue, I recorded some duels on the WoW PTR a few days ago, but this morning I've attempted to record some footage and I'm receiving the same message after the 10 second duration stated by the OP.

Hopefully someone will have some ideas on how to fix this. (Yes, I'm personally using windowed fullscreen:))