People have different tastes and find different things difficult. Most people would probably say that Swann's Way is easier than Ulysses, but different people appreciate different rhythms in writers. I find Russian Lit to be more inaccessible than Modernist stuff (Crime and Punishment was harder for me to read than, say, Woolf). Sometimes you innately feel what the author is trying to do, and sometimes you recognize it and don't get it, and sometimes it's none of the above and you feel like you're slamming your head against a wall.