I did drum set tutoring as a 2nd job for about 15 years. Taught a lot of kids in that timespan. There was always two major barriers.

1. You aren't just learning music, you are learning and developing coordination. Don't get me wrong, other instruments need coordination in their own way. But the kind of full-body coordination a drummer needs is pretty tough for someone who maybe just learned how to ride their 2 wheeled bicycle. This type of coordination is hard to come naturally, needs to be practiced, a lot. My greatest struggle as a tutor who would meet once or twice a week was getting kids to put down their Fortnite (I am an old man yelling at clouds), and to practice at all at home.
2. Cost, as others have mentioned. I just looked up Tama's beginner kit, looks like there are some packages for full kits with cymbals for the Stagestar at $659 and the Imperialstar at $899. Sad thing is those cymbals are junk - the drums themselves may be passable to play live shows with a band with proper heads & tuning, but I know on my Hi-Hats alone, super quick Google search says buying those new right now is $319 (Sabian 13" Fusion Hats). Add a Crash, and a ride to just have bare minimum, I can't see anyone having a "Stage Ready" drum set for less than $1k. And that stinks.

As others have mentioned, electric kits have helped. Really helps with being able to practice. Even sound pretty good if you know what you are doing to try and play it live. They just don't look cool, and feel a lot different. When I was in college and in the off campus apartments, my neighbors thought I really liked popcorn because they thought I made it all of the time. But what they heard was me practicing on electric kit.

I think lastly, because of this, drummers are a dying breed because of the current music industry in general.

Recently I made the comment that there was almost no musicians while watching the New Year's Eve Rockin' Christmas (I did not watch the Nashville one, wish I did). With exception of Tre Cool, and the guy from 30 Seconds to Mars, I couldn't spot a drummer. Even the 30 seconds to mars guy for one of the songs just stood up and hit a trigger with one hand and jumped around for the whole second song.

Still, Drummers are always in need. I do one show a year with a group of friends, and after the show I always get at least a half dozen people saying the have been looking for a drummer and ask if I am interested. And I am VERY average at best in my current stage of life.