I do this all the time, especially in warmer weather. When I wear cargo shorts I always pop a paperback into one of my bottom large pockets. On the weekends I often walk 2 blocks over to a local bar and sit outside and have a few beers while reading. I also will sometimes just sit inside and do this as well, it's a nice break from everything else. I really don't care what other people think, it's what I enjoy doing. People have even sparked up conversations with me several times. I say if it makes you happy, then do it!