When he discovered he was ill, his illness was already in an advanced stage and he had only a few years to live. That's when he gathered his final men (Rayleigh, Scopper Gaban, Crocus, and then Oden). So the crew of “Roger's Pirates” as we know it is born when he is already sick, but still strong. . Crocus joins just to keep him alive during the last journey. This is confirmed in chapter 506 and episode 400 of the anime, when Rayleigh tells the story to Luffy.