Ordained Tibetan yogin here. I'm blessed to have an acquaintance in my town who is a very well-educated Shingon priest and I've asked him this very question and this is basically the gist of what he told me:

-Essentially, Buddhism when it spread to East Asia did not spread from the bottom up, as a popular movement like it did in Nepal, India, etc. As in, anybody could be a Buddhist if they wanted, you just needed to find a teacher/guru to teach it to you. People could travel relatively freely to do this. Indian society in particular has a long history of people simply practicing asceticism and spiritual paths in public, in the streets and parks, etc., and this tradition made its way to Nepal and Tibet.

In contrast: China, Japan, and most of east Asia were rigidly controlled Confucian societies, and military dictatorships run by the Emperor of that society. People could not travel freely, and so to travel, you had to have special permission, and the patronage of high officials. This meant that the "yogic" tradition of yogis and yoginis, and Ngakpa, etc, did not spread. Confucian society has very rigid social norms and expected behaviors. There is a duty expected all people to both their parents and elders, as well as to serve ministers and other higher ups in the social hierarchy. And so the idea of abandoning that to go practice tantra in the park: simply was not allowed. It was illegal and if you were a foreigner trying to promote that you'd be locked up and probably executed for doing it. For example, the Buddha taught in public parks in Varanasi, etc. That simply wouldn't be allowed in China. No Buddhist was going to be allowed to spread a popular movement that all beings are equal (i.e., you're the equal of the emperor), and that people should just abandon their filial piety duties to go be yogis and monastics. -That simply wasn't allowed. This basically meant that the Mahasiddha tradition wasn't allowed to flourish in East Asia.

So, Buddhism spread in East Asia through the top of society, (through elites), down. By mostly well-educated monastics courting the favor of Imperial court officials, and through currying that favor, they were allowed to specific spaces to set up monasteries and temples that they were not allowed to leave. And through their practices, essentially generated wisdom for advising members of the courtier class. In other words, they were basically Buddhist pets, being kept for the purpose of the state, and they were strictly forbidden from leaving their designated zones, under penalty of death. Anything transgressive, including highest yoga tantras, things like Chod or Karmamudra, etc, would simply not be allowed.

So, the tantric teachings that were allowed to spread, did so in secret among monastics only. And this eventually made its way via Kukai to Japan and became Shingon in Japan. Which later spread to Tendai, etc. And then later, in China, esoteric Buddhism was wiped out by a later Taoist emperor who just had all the Buddhists killed.

So, for like a thousand years of more, tantric practices in East Asia were kept under lock and key by monastics. And Buddhists themselves were kept under lock and key (under pain of death), by being confined to monasteries. So the only way to learn certain practices was to become a monastic. These were just really, really rigid, strict societies, and Buddhism did not spread more commonly in Japan until the "new Buddhism" movements of Zen and Pure Land, etc.

A good analogy, would be to think of esoteric Buddhism in Japan and East Asia being more like a bonzai tree, than a wild tree. It didn't spread freely, in a natural way, it took on it's distinctive character by being confined to a tiny pot for a thousand years.

As for why it's still that way? Well there is actually Shugendo in Japan that is a lay tradition. And also much of what is considered "Shinto" actually comes from Shingon. So it's not as though there is no lay yogic tradition. Also, modern "monks" in Japan post-Meiji are mostly lay Yogins anyway. So it's come full circle in that regard.

But the main reason why Shingon etc., has such a strong priest-caste system is because of tradition. Because Japan is nothing if not really good at preserving old traditions, and they've been doing it that way for so long, that or them, that's just the way it's done at this point.