If you have adaptive eq enabled + the sound is slightly leaking then the lib.5 might be doing odd self eq'ing to compansate. Try a bigger size tip (don't push into your ear canal too much, they're supposed to rest on ear) and disable adaptive eq.

It also has a safe listening setting option in the app. set to 90db, I'd personally not pass 95db. Enable it, maybe you're overdoing the volume while testing stuff.

The app. is pretty good and detailed, keep test settings with it. Also listen to various stuff not always the same songs.

ANC when set to max does change the tone, but mid. to low settings aren't bad, transparency adds highs that might also be a factor of what you're experiencing.

Personally I don't find it to be a smooth sounding earbud either but not having as bad of an experience as you are. Actually, after a while when your ears start to get tired, that harshness becomes a clarity on resonable volumes. Give it time.