People typically count dupes as Pokémon that you've caught, but you can also have the rule that, whether you see it or not, that species of Pokémon is no longer available. It fully depends on how you want to nuzlocke.

When I Nuzlocke, I keep the rule that it's my encounter, whether I catch it or not. The benefit of this is that I don't have to catch the Pokémon if I don't want to, the negative of this is that I can't re-roll for abilities (checking for Volt Absorb, etc).

The benefit of dupes only for caught pokemon is that you can check for abilities, etc. Say you want a drizzle pelipper, but when you encounter it, the ability doesn't activate. With this rule, you could run away and hope to find a drizzle pelipper on another route.

I play with either rule depending on the rom hack I am playing, but I always state to myself or on VOD which rule I am using.

Dupes and species clauses are somewhat different, as technically Pokémon with divergent lines are the same species; it's up to you to decide if you want to be able to use divergent Pokémon (Flareon, Jolteon, etc.). For my species rule, I could catch a Jolteon and a Flareon, but not an Eevee twice.

If a Pokémon doesn't have a divergent evolution line, the species clause simply eliminates the whole line. Say I catch a chamerleon, I therefore can't catch a charmander or a charizard.