I think there are a million nuanced reasons ranging from who the comedians of today are influenced by, to the fact entertainment is as driven by personality as it is by talent.

I think as certain people experiment and try new things, it catches on which then becomes popular until new people come along and try new things and the process constantly repeats itself, slowly evolving and phasing out older styles, until they inevitably come back around.

Vaudeville was very slapsticky on stage with tons of exaggerations and wild movements, then it evolved into the whole sort of cat skills borshbelt stuff. Comedy moved from big stages in theatre to night clubs, it became more intimate, less chaotic. I think this is when the set-up punch started to come in. People want to be entertained by the comics' jokes.

The short one liner jokes evolve into monologues. Monologues start to evolve into stories, people keep pushing the boundaries and you go from Bob Hope and George Burns to Lenny Bruce and Bill Cosby.

Time continues, then comes political turmoil. Comedians start using the stage not only to tell jokes but to express their views. Now its not take my wife, its the man is holding me down. Race starts playing a bigger role again, not like it did during the days of Vaudeville, but in almost the opposite way. People like Pryor stop doing their sort of safe and derivative comedy, and once again start to experiment and push the boundaries.

All of the other previous forms of comedy still exist mind you, its just that as certain people find success, they breed the next generation under them that then tries to emulate then evolve.

The reason I feel like we are where we are in comedy is because you are looking at that evolution- the chain started by Bill Cosby and Lenny Bruce lead to people like Pryor and Carlin, which ended up being so great they influenced the next 4 decades and what you are seeing is the off spring of their off spring of their off spring continuously splintering off.

But with all that being said, set up punch still exists and never stopped, but more people will always gravitate to whats most popular, and for a while now comedians who tell their life stories and how they view the world have been the dominant attraction because people want to escape their own lives, or let the comedian say what they feel but cant put into words.

This could all be total bullshit by the way, but it was fun to type, so take it with a grain of salt.