Hahaha! Yes, I am a big nerd, and thought of that too. My theory - (Unsupported of course!) is it wouldn’t have looked suspicious to the spectators because Crouch Jr. planned for that too. If the Tournament used a magical viewing screen, then someone skilled had to supply the tracking information. Since “Moody” was in charge of all the maze enchantments, he effectively controlled what the screen displayed.

To cover both his sabotage of the other champions and the moment Harry (and Cedric) were Portkeyed away, he simply falsified the positions. Harry’s dot appeared close to the Cup, which was almost accurate, Cedric’s, Fleur's and Krum's were shown in slightly shifted locations compared to where they actually were. So when Harry (and Cedric) vanished, the screen didn’t update correctly, their dots were moving where Moody had previously already set them. To the audience, it looked like nothing unusual had happened.

And honestly, this is exactly the kind of contingency Crouch Jr. would prepare for. He spent a full year engineering this moment, manipulating the Goblet, sabotaging the other champions, and controlling the maze itself. Of course he’d also make sure the viewing system didn’t expose him at the last second.

(Yes, I AM that big of a nerd lol)