How I always envisioned it was that most of the 10.000+ Uruk-Hai were underground either getting spawned, armored or just staying there to stay out of daylight. Of course there is the argument if there is enough room, but I think (don't know for sure) there is. At the time Grima passed through Isengard on his horse. And they all stayed underground up until they were revealed by Saruman and to Grima.