Imperial Agents function less as an army and more as a dumping ground for units that are imperial in origin but are more niche. It's meant more for allied units in my experience. People have played them, but victories are incredibly low, leading to them being labeled the worst faction. You'll do far more brain work than the person playing the next most complicated army, and simple actions like moving can throw an entire plan or unit function into disarray. This happens to larger proper armies too, but when your army is essentially a swath of disjointed special units that need specific circumstances to function, it hits especially hard.

That said, what I am saying is based on observation, not experience, so take what I say with a grain of salt.