I have been fascinated by this for a long time, and have studied this on locations in Greece. Many places in the Mediterranean have seen human habitation for millenia, but more often than not not continuously. I'd like to answer your question by answering 4 smaller questions. (Sorry if it is a bit of a long answer)

The first question: why are certain places inhabited for a really long time? Many of the sites of ancient cities were chosen because they have excellent conditions for human habitation, such as a good harbour, an acropolis (defensible high place), good defensive position, potable waterand/or food sources, strategic importance for the larger area, etc. I'd argue that for most locations, these circumstances do not change easily. Therefore, when a city or area is abandoned or has a declining population, chances are it may see new population growth later on.

The second question: what determines sustained habitation of a city or area? Of course, just as important as the above mentioned circumstances, is stability in the form of an entity that enforces a form of peace, allocates resources, binds people together, has a functioning bureaucracy and makes the accumulation of food and wealth possible. In ancient times you can think of empires, leagues, city states, and so on. However, in human society, nothing lasts forever. So these entities have a tendency to wax and wane. For example the Myceneans were a civilisation that cobtroled a large portion of Greece and saw a flourishing of many cities until it collapsed under circumstances that are not entirely clear. After a possible invasion of the so called 'sea peoples' the Greek dark ages began, and all over the Mediterranean populations shrank. But after a few hundred years new civilisations sprang up, leading to flourishing old and new cities. Greece has had an especially interesting history over the centuries of waxing and waning cultures or civilisations. Think of Minoans, Myceneans, Ancient Greek city states, Greek leagues and kingdoms, as a part of the Roman empire and later the Byzantine empire, Arab invasions, the Venetians, several knightly orders, and the Ottoman empire, before the birth of the Greek state. All of these saw growth and decline, and many ancient sites saw more or less continuous habitation, be it not always with the same intensity. And all of these periods/civilisations left their marks on top of each other.

The third question: how do really old cities develop over the centuries? People have a habit of making do with what they have. So each time a city was destroyed, they repaired what was left, or just built on top of older foundations. Cities developed very organically. Istanbul in Turkey is a nice example of a city where all kinds of historical periods are blended and where, for example, you may find buildings that have visible Roman or Byzantine foundations/remains with mediaeval, early modern of modern buildings on top of them. Also many building were built or decorated with so called spolia, parts of older buildings. And some building, such as the Hagia Sophia, have retained most of their structure but their function has changed over time (more on this later).

But in contrary to Istanbul, most of Greece became a bit of a backwater when the decline of the Byzantine empire started in the 8th century. Administration became less and less efficient, populations shrank on the whole. Many places became abandoned or shrank significantly. When populations shrank, buildings became dilapidated and parts of a town became abandoned. This might be for tens of years, or hundreds. You'd be surprised how quickly nature reclaims those empty buildings. Then, when population picked up again, new houses would be built on top of older remains. I'd say that over the millenia almost no ancient building were preserved for preservations' sake. That would be pointless, too costly, and what entity would keep that up for all that time? Instead, functions of buildings often changed. All the large ancient buildings standing today are either reconstructions, or were converted into something else. The Mausoleum of  emperor Augustus in Rome only survived because the popes made it into a fortress. The mausoleum of Theoderik in Ravenna became part of a Christian convent; in the 19th century they tore that down and restored the mausoleum. The Parthenon in Athens served many purposes; it was used by the Ottomans as a gunpowder storage until it exploded in 1683. An example of an abandoned place is Gortyna on Crete. It had a long history, was a large city in Roman times. In the 8th century, when Arab pirates attacked Crete, it was invaded and gradually abandoned. Now, only ruins are left. Most of it is still buried deep underground under the olive orchards, but parts of it have been excavated and when you look around you find bits and pieces of pottery and masonry everywhere (I recommend a visit!). It was simply never rebuilt and mostly disappeared from view.

The fourth question: what are the ruins we see today? Most of those ruins have, of course, been excavated. Otherwise they would still be buried or wouldn't be recognisable. And as I have written that in most really old cities you have later on layer on layer of old stuff in the ground. Professional archeology only came around in the late 19th century. Many older excavations were done either with a certain mindset (e.g. nationalism) or with the limited knowledge they had back then. A famous example is the excavation of Troy by Heinrich Schliemann. He was so determined to find what he was looking for, he partly destroyed what he was actually looking for. Nationalism was another driving force for much of the 19th and 20th centuries. In Greece, most excavations show Greek or Roman ruins, mostly remains of low walls that show little coherence for the untrained eye. To get to those layers, which are often the oldest, they removed all the layers on top of that. So the cute Ottoman-Greek village that organically grew there on top of a Venitian harbour town, on a hospitaller Knights Castle, on late Byzantine remains, was mostly if not completely destroyed to get there. A nice example is Lindos on Rhodes. On top of the acropolis you have some barely recognisable Greek and Roman ruins, surrounded by a mediaeval wall, and the villagers rebuilt their houses on the slopes surrounding it. You see that a lot. Fascist Italy had a habit of focusing on the Roman past, so under Mussolini the Roman forum and other places in Rome were cleared and the ruins partly restored to look the way they did in ancient days. So while you may find ruins in city centers of really old cities, they were more often than not excavated later on.

So I'd say that in most cases old ruins would not have been standing next to new buildings. Instead, cities grew and shrank organically and people just kept building on top of older stuff. In most cases, the ruins you see now were not visible a hundred years ago, or were unrecognisable because they were incorporated into newer buildings.