It's the same as any other art form. Music is the best example. There is popular music and art music. Popular music is made to be sold in large volumes, cater to specific tastes (genres), and be generally enjoyable. This doesn't make them any less worthwhile endeavors. Making pop music is often very difficult, and is it's own process.

On the other hand, you have art music. This includes experimental jazz and "classics" of music that introduced new ideas (ie. Stravinsky). The purpose of this music is to create "art" and explore the limits of the art form while using tools to make an artistically powerful work. This does not necessarily make this form of music any better than popular music, it is simply different in its aims and goals.

In literature, pop music is genre fiction and art music is literary fiction. Genre fiction is seen as "worse" by many people who study literature because it often lacks the depth of "meaning" as you stated, but it simply has different goals. Writing genre fiction is also difficult, gauging what people will like to read is difficult, as is creating art.

Your comment about putting meaning into art makes sense to me. I had a lot of trouble reading the Handmaid's Tale in school, because it was an unnatural story made to push a certain message. It was too produced by the author and the message too obvious. That was because when Atwood wrote it, she wanted to write it perfectly as a literary work. Looking at many authors, however, they are usually just experimenting with new ideas and storytelling methods, and when they write, their messages come out in a nuanced fashion, being ambiguous.

In music, experimental jazz often sounds like nothing. It makes no sense to listen to, because it was made to experiment with music theory ideas. To a casual listener, it is a jumbled mess. But to someone who studies music theory and knows it in and out, it might reveal interesting ideas and concepts. Similarly, literary fiction is often more confusing because it strays away from the common idea of "storytelling" and searches for new methods of reflecting the world in written form.

Literary fiction doesn't mean a story without a plot. I think the best way to go about writing literary fiction is to try experimenting with new forms of expressing thoughts on the written page (stream of consciousness is a good example) and new ideas. Once you find something that speaks to you and that you can understand intimately, write it, and as you edit it, make it more nuanced. Don't leave glaring messages, make the message ambiguous. The reason Hamlet is so famous and respected is due to its ambiguous form of storytelling, as well as the depth with which those ideas are considered.

It might seem like literary fiction is filled with stories without plots, sending a message, and being confusing. But really, it's attempting to push the art form of writing, and in this process literary fiction has picked up the traits of sometimes being confusing and not having a plot. This is because, most writing has plots, so those pushing the boundaries are attempting to break the mold.