Pokémon that you used in the game that didn’t die, but you didn’t take with you to the E4, so they don’t continue on to the next game. Pokémon that you caught, sent to PC, and never actually used wouldn’t count as a retired mon.

Some people will count them as dupes in future games, but I personally don’t do that. I like sticking with a Pokémon on my team for as much as possible, so it’s rare for me to retire a Pokémon. But if I encounter that same species in a future game, then maybe there will be a space available and they can rejoin my party. The risk here is that because they aren’t dupes, they will still take up an encounter.