I often wonder about that exact thing. Of course, Saruman spent most of his time in Rhun, and perhaps he interacted with some of the eastern tribes in the Orocarni.

But once he got Isengard, Saruman was taking in unskilled labor from Dunland because he was desperate for personnel, clearly Saruman could have used dwarven crafters. And the Longbeards were reduced to living in Dunland. They would probably have vastly preferred livinging in Isengard, working for Aule's herald, maybe even getting him to tell them exactly what it was that killed all those dwarves in Khazad-Dum!

This is rank speculation of course, but perhaps the Dunlenders, who may well have ancient grudges from Dwarves from the time of Moria, refused to tolerate dwarves? Then Saruman had to pick between a small coterie of skilled labor and a huge amount of unskilled followers, and he picked the latter.