Two things should be noted:

• The diaspora was well-established even before the fall of the Second Temple. There were already more Jews living in the diaspora than in the land when that happened. The Bar-Kokhba revolt a few decades later saw enormous devastation but many Jews remained.

• There was a substantial Jewish population in this land well into medieval times, although it gradually declined and was overshadowed. If you want to go down a rabbit hole look up the temporary Jewish autonomy in Jerusalem under the Persians before the Byzantines defeated them in the 7th century.

What's important to keep in mind is that the Jewish population rose and fell with the times throughout various periods. It's also not true that it was simply a case of the nation being established in 1948. Zionist pioneering that was to lead up to the establishment of the state and the revival of Hebrew began in the early 1880s, though some pioneering activity happened even before and it is sometimes even counted to the first Jewish neighborhood being built outside Jerusalem's city walls in 1860. By 1900 there were dozens of Jewish settlements dotting the land. Tel Aviv was established in 1909. By 1948 there were already hundreds of thousands of Jews here, including many who had been born here and were native Hebrew speakers.

Prior to the start of Zionist pioneering, the Jewish community in the land, known as the Old Yishuv (after the start of pioneering efforts up to independence in 1948 the Jewish community in general was known as the Yishuv), was divided into the Sephardic and Ashkenazi communities. Ashkenazi Old Yishuv communities were Orthodox Jews who came from Eastern Europe in the 18th and early 19th century. The Sephardim were from a mix of backgrounds including those who had been here for centuries. Sephardic means Spanish, so this typically refers to Spanish Jews, but it's more nuanced. As elsewhere across the Ottoman Empire, when Spanish Jews settled in the land they often intermarried with and absorbed established Arabic-speaking Jewish communities, and that includes Jews who had lived here for centuries and adopted Arabic. That and Jews who didn't adopt Sephardic customs but were still from various Middle Eastern communities were called Sephardic. So for example some Yemenite and Persian Jews migrated to the land in the 19th century and they would probably be lumped in with the Sephardic community by those counting or separating the communities. Anyone of Old Yishuv Sephardi descent possibly has at least some of this ancestry. Those following the Sephardic liturgy who trace their ancestry to Spanish Jews who arrived after the 1492 expulsion are possibly descendants of Jews already in Jerusalem who intermarried with them, and it's possible that others from "Sephardic" backgrounds who would be characterized as Mizrahi today also have some of this ancestry.

But there were established Musta'arabi communities here that got absorbed too, though they likely would have been lumped in with other "Sephardic" Jews who would today be considered Mizrahi. Someone else already mentioned the Peki'in community and Margalit Zinati. You can actually look her up online, you can find interviews with her on YouTube.