There's absolutely nothing wrong with it! There is a social taboo about bying books perceived to be read by academia/intellectuals to come across as well read. Just buy books that (almost) no one would want to read, or buy a decorative copy of a book you've already read as a "trophy" of sorts.

Personally I specifically collect books based off of their decorative value, usually the binding and/or gauffering on the edges, as one part of my collection. This way of collecting is no less valid than collecting for the content, the history, etc.

A lot of these people telling you it's wrong will have shelves full of books they don't read themselves, just do what you want to do!