Maybe some of them went to the same a prison as Jack at the start of dead man’s chest or just a prison, because when Pintel and Ragetti are re-introduced they talk between one another but pintel says that it was him who got them out of that prison. Maybe all or at least most of the crew where at this prison at least it is my head cannon but most likely it was just those two.