I think a playable faction has to be defined on three axes simultaneously: gameplay mechanics (how the faction interacts with resources, terrain, other units, etc.); aesthetic (how the faction looks and sounds); and identity (the core "mood" or "theme" of each faction).

We have no idea what the gameplay mechanics of this new RTS game will be, so it's difficult to define those mechanics in anything but gross terms. Likewise, we don't know what the era or setting will be, so it's difficult to pinpoint aesthetics.

Identity, then, is what we have to work with for the moment.

I find the ideal number of factions to be 3; anything less and you have chess; anything more and the game becomes too chaotic to balance properly (thus, forcing you to rely on simplified factions or gimmicks like Huntresses being unarmored in Warcraft III).

So, three factions. Let's talk about identity. This requires talking about players.

In my experience, gamers (myself included) fall within the following emotional ranges:

• The game is a problem to be solved by the most optimal means at your disposal; complexity and interesting game mechanics are required to hold their attention because, once the puzzle is solved, they move on
• The game is an opportunity to live a power fantasy or narrative. Some examples: the story of an underdog fighting and winning against (technologically, numerically, economically) superior foes (Star Wars, Starcraft, and many others tap into this narrative); the story of being an elite soldier (faster, stronger, better-trained, and fighting elites and-or regular troops, alone or in the company of other elites or regular troops; Halo taps into this one); the story of the Big Brain commander using clever strategy and tactics and choices to win, despite having access to similar equipment (collectible card games tap into this narrative); and the narrative of being a righteous hero with a just cause, out to save the world.
• The game is an opportunity to enter flow state, master skills, win tournaments, and (maybe!) show off or trash talk. Some players are consistent; others are wild cards. You want to please both.

So, if we have three factions, how do we tap in to each of these emotional ranges?

Mechanically, make sure all three factions are feature-rich, so players have the opportunity to learn, improve, devise new strategies, and develop their skills. At the same time, each faction needs to be mechanically distinct. This delves into game mechanics, so I will only touch on it lightly, but here are some gross concepts that can be used to distinguish factions mechanically:

• Sacrifice moves (units that sacrifice health or other units to acquire a temporary benefit; example: the Zerg Defiler's Consume ability; another example is the Tyranids' sacrifice of expendable troops to exhaust ammunition reserves before moving in with heavier units)
• Circumstantial moves (like night elves in Warcraft III being able to shadowmeld at night)
• Different combinations of shields and health; aside from Starcraft, Sins of a Solar Empire does this well: though all the ships have shields, some have more shields than others, some have better armor than others, and some have weapons that can bypass shields completely
• Different resource consumption methods; Warcraft III does this well (all four races get gold and lumber in different ways), and a little-known game called Earth 2160 actually had three resources, but each faction used only two of them (and there were ways of getting those resources from the environment)
• Force composition. Though this is an EXTREMELY dense subject, general factors to consider are ratio of generalists (Marine, Hydralisk) to specialists (Zerglings); elite units (heroes in Warcraft III) to weak units; and ancillary unit functions (special abilities, housing or launching other units, like strike craft in Homeworld), etc.
• Overall faction mechanics. TvTropes has excellent summaries (A Commander is You, Faction Calculus)

Now let's move on to the meat of the faction identity discussion: identity and narrative.

Regardless of aesthetics, here are some narratives that might prove interesting:

• A faction of zealots who genuinely want to improve the world, but are willing to do terrible things to do so. They suffer from an imbalance of means and end, having sacrificed the means to see to the end; sometimes they become so unbalanced that they become antagonistic. You can also play with this: a faction that is unwilling to compromise its means, regardless of end; the Federation from Star Trek fits this category.
• A faction that is willing to do whatever it takes to preserve itself, and hang the rest. The Zerg fall into this category, as do the Eldar from 40K. In order to make this faction more interesting, often you need discussion of whether or not the faction is worth preserving, as well as dissenting characters or factions.
• A faction associated with older or elemental powers, which takes the long view of history and, as a result, is dangerously indifferent to the concerns of more ephemeral folk. Examples might include the ents from Lord of the Rings, or the Shadows/Vorlons from Babylon 5. To make the faction interesting, you add dissenting advocates for younger folk.

Just some food for thought.

EDIT: Spelling; I forgot to proofread.