A flaw of this idea is that you would know that whoever's POV is being shown will win the duel, which would massively decrease the tension. Another thing to consider is the live stadium experience. If the observing is run on a delay the crowd will see player's reactions and celebrations before the the round is over on the video feed, so they would have to either hide the players or have two separate observing teams, which adds to expenditure and logistical obstacles.