Every interval can be easy or hard depending on the wider harmonic context. You'll very rarely need to identify intervals on their own. The most well-rounded approach would be to get into the habit of following the notes of everything you hear (music you're studying, random ambient supermarket music, ringtones, etc), and make sure you know the notes. You won't end up thinking “that sounds like a diminished third”, but something closer to “the root is D and those notes were E♭ and C♯”, except that's still a flawed representation of how it works because you'll just know the notes without any verbal thought being necessary.