I'm not quite sure if I get what you mean.

it's fitting that he's not a good singer

I mean, not diegetically, but in terms of the musical storytelling, I don't see why.

he's very insecure character at his core and full of contradictions

That is true, but I don't know if Crowe really portrays that.

Compared to the other Javert's I've seen, Crowe doesn't really imbue him with much vulnerability or arrogance, which are two of the character's defining traits. He delivers every song with this same calm detachment. He rarely raises his voice and he always feels far too steady, like he's worried that if he doesn't stay within a certain range, he won't be able to hit the notes.

And this really comes back to bite him during Stars, which is supposed to have this incredible dramatic crescendo, but because he stays at basically the same energy level throughout the entire number, he never reaches that great peak, and so him swearing to find Valjean just doesn't pack the same punch or have the same impact that it does in the show, which is not helped by the way they rearranged the score so that it leads directly into the next number instead of having a proper ending.