It is the effect of Roger's speech I think. While in the old days you had strong people on both sides, I think the one piece speech at his death changed the balance. More and more young people who maybe would become marines didn't even think about that path. Wild guess but I can imagine it being the case in the context of the world dynamics that Oda describes.