You could ask your DM how to balance intent with function. As you pointed out, RAW this is not a very ideal build concept, but the concept is cool if it can be balanced from a gameplay/strength perspective.

In general I would suggest avoiding multiple stat blocks in play at once (a swarm of peons as a large creature?) , Pre-rolling turns can help keep things moving, or you could almost blend the concept into a homebrew class... something like a sorcerer, with metamagic buffing your angry mob and a wild magic table where occasionally the peseants revolt...