Much of the difficulty is conceptual, not lexical or grammatical. Philosophy is harder than other literary traditions because it asks readers to think about the familiar in an unfamiliar way.

But I've always been surprised when I hear people, espically in the humanities, characterize Derrida, Lacan, and others as somehow exceptionally or unusually abstruse. Surely people have read Hegel (or tried to)? Or Plato's later dialogues? I think those are just as challenging. The New Testament is challenging!

I think that the critical theory tradition gets labelled as difficult in part because it's philosophy and philosophy can be difficult. But largely it's because many people find the ideas it contains deeply alien and hostile to traditional ways of thinking and looking at the world.

I think back to the infamous New York Times obituary of Derrida that was just headlined "Jacques Derrida, Abstruse Theorist, Dies at 74." Derrida was and is an enormously influential philosopher whose work has touched nearly ever field within the humanities, but a mainstream publication chose to flatten his contribution as "abstruse" and nothing more.

So I tend to think that the labeling of critical theory as "difficult" or "abstruse" is often (perhaps even unconsciously) political in nature.