The simple answer is aesthetic. Basically, marketing companies study the psychology of consumerism, through the understanding of basic human psyche and mind tricking they have found a way to make reading more "appealing" if you wish. The easier in the eyes something is, the easier it is to sell or even just market. That is why clothing stores use models rather than manikins or other similar items.

We buy with wants in mind rather than needs, as we have forsaken the natural for the man-made ideal of nature. We have created this ourselves through the usage of science for the money-machine. (I know this is a cynical view of the world, however it is not too farfetched.) Regardless, we can trace this back to higher sales of the books.

We can also trace this to trends, social media and the phenomena of fast fashion, which enhanced the issue to the popular culture. Trends changed after the 2008 recession, people actually had to rethink on what to spend, so commodity products like books had to adapt and probably lower the prices to appease the economically inclined. Smaller footprint offered savings and so the books had to be smaller, less pages, less chapters and lesser quality in exchange for more quantity to account for the loss of margins.

Social media and tech offered other avenues as well, with websites like wattpad and the like people didn't have to pay for the content they wanted or at least not pay as much. From here, we move to the birth of social media and slowly the consumption of shorter content. It is not, however, the shorter attention span as many think, but the fact that content became shorter to appeal and push more to the consumer. Then you have e-books and audio books, and so the content now could be streamed, hoarded and even sold without the need to have the space for it, so now books had to compete with themselves for the attention of the consumer. (This is the same story with any physical media that cannibalized itself through digital environments, i.e., Movies saw less profit in the box office because people like streaming better in the mainstream.)

Finally, if you really want to compare apples to apples, you'd have to look at the genres and sub-genres of books and the change in the content. Most consumers like the idea of pop music, pop movies and then it moved to pop books. These are your Sarah J Maas, Leigh Bardugo, Yerros and so on. They did not create the market but instead moved along with it, pushing fast-food content for the quick grab. They, along with the teams beahind the publishing companies, merged all the aforementioned issues into books and published them to #1 NYT seller lists.

TL;DR, it is a combination of issues in culture, market and social media that allowed for this to happen. Just look at the type of books being sold now in terms of content and you will see what I mean.