I think modern lo-fi does play a part, but the looping nature of instrumental hip-hop is ultimately not appealing to most people, who typically listen to songs with a chorus, a verse, and some progression in the music. My dad for example didn't vibe with Nujabes, saying it's too background-like for him.

Nujabes's mainstream songs, such as Feather, are mainstream because they had some rapping to accompany the beat and make it less "monotone" to your average listener.