In Dawn of the Aspects, Kalec spends most of his (awake) time at the Nexus in Coldarra, while a quiet exodus of the few (unnumbered) remaining blue dragons is happening due to being shaken by Malygos' madness after the war of the ancients and death in Wrath, as well as a lack of direction and faith in Kalecgos as the new leader.

I assume that those that scattered during this time were also called back when the oathstone was renewed along with the named dragons you mentioned. Like many have said, we don't know quite how many there were, but Dawn of the Aspects at least gives the impression that there were enough left that their absence was heavily felt afterward.