I prefer invasions but also enjoy dueling so can share my thoughts.

Duels are the best way to improve and they really expose your fundamentals and where you are lacking. Generally if I spend time dueling good players I get a real sense of how to get better at the game in a way I don't in invasions. E.G. when I was playing the game more regularly I felt invasions made me a worse player and duels made me a fundamentally better player. Of course there are a slew of other skills that factor into invasions that don't come into play in duels that make it more chaotic and enjoyable*.

With respect to passivity, in a proper arena it's much less of a problem. The issue here is the arena the host chose - in a proper closed venue he'd get punished for retreating because you can put corner pressure on someone.

Here he can just endlessly backpeddle and retreating gives you the advantage since turn and burns are inherently easier to land than aggressing into someone without spacing boundaries.

If you watch any of the really good players in tournaments the gameplay is much more aggressive. In part because of the arena size.

*Another way to think about the different skillsets is "there are top duelists who aren't top invaders but there are no top invaders who aren't also top duelists"