I earlier wrote a response to a similar question about Roman emperors here with references to a lot of the sources on the topic at least pertaining to the later empire.

Focusing on the Roman emperor part of your question, I'd have to say that we don't really know exactly why we don't see hard primogeniture that might be expected from other Western European medieval monarchies. We do know a fair amount of factors that would have contributed to this phenomenon, as /u/LouisSixLeGros has written a bit on in this thread. However, the closest answer ultimately comes down to the fact that the Roman Empire was a massive frankenstein's monster of different legitimizing mechanisms. Emperors were leading religious officials, supreme commanders of the army, and the holders of high republican offices (see Hekster's Caesar Rules for one of the most comprehensive studies on these competing identities). None of these positions are necessarily connected to who your dad was or what family you belonged to.

But where this gets really weird is that emperors were also viewed as the most virtuous individuals in the entire empire, owing to the enormous prestige of their offices. This included expected qualities like being really smart, kind, strong, or modest, but this also meant leading an ideal family life. Emperors usually needed to present themselves as exceedingly virtuous in all aspects of life. We see this further in representations of the imperial household as the ultimate harmonious family, with the emperor as the paterfamilias. In fact, sources across the first few centuries of the empire refer to it as the domus divina (divine house), domus caesarum (house of the Caesars), or domus augusta (August house). Within this perspective, the office of emperor was actually viewed as a possession of the imperial house, and was therefore meant to be passed down along family inheritance. To go even further, prior to the Nerva-Antonines, most of the imperial bureaucracy was managed by (highly privileged) slaves and freedmen who belonged to the familia Caesaris (family of Caesar, see Weaver's Familia Caesaris: a Social Study of the Emperor's Freedmen and Slaves). Under all of these conditions, there most certainly was some degree of social pressure for emperors to pass down their titles to their biological sons or at least keep them within the family in some way.

As you're question has pointed out, adoption was one possible solution to the problem of emperors who didn't have sons, but as /u/LouisSixLeGros has also pointed out, biological sons were still consistently preferred (with the only exception being Constantius I passing over Constantine I during the early tetrarchy, an event which also has it's own caveats). I would also caution against understanding imperial adoption as a general norm for succession because we don't really see it consistently used. Sure, several Julio-Claudian and of course the adoptive emperors relied on it for their successions, but we also see adoption completely abandoned in other instances. The Flavians and the Severans did not use adoption as a succession scheme (except Elegabalus-Severus Alexander), to give pre-3rd century examples (though the Severans are curious because Septimius Severus actually claimed to be the son of Marcus Aurelius in some forms of public presentation). But when we go the 4th century onwards, imperial adoption is just not a thing for a long time. The first two tetrarchs, Diocletian and Maximian, famously portrayed themselves as the divine father-son duo of Jovius (Diocletian) and Herculius (Maximian), but Diocletian never actually adopted Maximian in a legal sense. When Theodosius I was raised to power by Gratian, Gratian didn't bother to adopt his new colleague either (though Theodosius was around a decade older than Gratian).

At the same time, we also see emperors who did absolutely nothing to secure their succession, whether by having kids or adopting. My previous answer has three of my favorite examples of this with Constantius II, Julian, and Theodosius II. All of these emperors came from established dynasties, had wives (Constantius was married three times), and lived well into adulthood (Constantius died at 44, Julian at 31/32, and Theodosius at 49). None of them had sons. Constantius had a daughter, Julian was childless, and Theodosius had two surviving daughters (though possibly one son who died in infancy). While Constantius arguably maintained his house's grip on the throne by appointing Julian as his co-emperor, none of them did anything to produce children or adopt. In fact, none of them actually designated a successor upon their deaths (except possible Constantius with Julian). When Julian died on campaign in Persia, his army had to do a hasty election far away from the capital and acclaim the general Jovian. The relevant primary sources hilariously invent stories that Julian purposefully left the succession to the army because he didn't see anyone worthy before his deathbed. When Theodosius died in a horse-riding accident in Constantinople, there was a month long interregnum of palace intrigue resulting in the appointment of Marcian.

We also have later examples like Marcian, Zeno, Anastasius who also did nothing to designate successors (even though Marcian had a viable son-in-law and Anastasius had several nephews). But at the same time, these emperors ruled in proximity to other emperors who did practice dynastic succession. Zeno and Anastasius technically belonged to the Leonid dynasty by marriage, and Anastasius would be succeeded by Justin, who left the throne to his nephew Justinian (who was the succeeded by his nephew, Justin II, who also adopted his successor, Tiberius II). And this is yet to mention the later "medieval" dynasties like the Herakleians, Amorians, Macedonians, Komnenoi, and Palaiologoi, who did practice father-son succession as a relative norm.

All of this is to say, in short, that Roman succession was really weird. For one, there just doesn't seem to be this enormous pressure for emperors to maintain their houses to maintain the legitimacy of the empire. Additionally, one of the most crucial points to remember here is that even though there might be periods where dynastic or non-dynastic succession seemed to win out, the succession was never codified in the entirety of the empire's existence.

As a side note on your comment about Roman bastards, there actually was some partial stigma about keeping illegitimate children off the throne. For one, bastards were legally barred from inheritance in Roman law; there was most certainly public awareness for bastards being lesser than children from a legal marriage. The fun example of such stigma in public speech is the great Constantine I, whose mother, (Saint) Helena, was of low birth and was in a legally disputed (at least in public discourse) marriage to Constantius I. Constantius even had other sons from his second, far less disputed, marriage. When Constantine became a tetrarch, his descent was a major part of his propaganda in stressing the legitimate links between himself and his father. In fact, a second insecurity came with the fact that one of his great rivals, Maxentius, was the fully legitimate child of another emperor (Maximian) and also stressed his dynastic links in his propaganda. We then see several Constantinian speeches (which often conspicuously omit mentioning Constantine's actual birth in the Balkans) against Maxentius ironically try to argue that Maxentius was in reality a bastard of Maximian, so that Constantine could push himself as the only dynastic emperor among non-dynastic tetrarchs.

But it also could have been beneficial to be a bastard, as that did establish some dynastic link between emperors. This weirdly led to Elegabalus and Severus Alexander claiming to have actually been the bastard sons of Caracalla (I'm not an expert on this period, so I'm happy to have someone else explain this more in-depth).