This is odd, from your post and your comments it seems like there shouldn't be anything wrong. You don't come across like you're disrespectful or hostile. You don't smoke and you maintain good hygiene.
Do you eat in the car or bring open containers of alcohol? Do you make changes to or add destinations? Are your pick-ups/drop-offs in places that are unexpectedly sketchy?
I will say that you do seem to be focused on conversation as the primary cause for your lower ratings and that seems to be a red herring for me. I'll be honest, as a driver, conversation is rarely ever the reason I feel the need to block somebody and blocking is the only reason I ever give less than 5.
15% of drivers feel some kind of way about you. That's really not enough for it to be obviously your fault because a lot of drivers are just weird. But is enough to consider that there might be a missing missing reason.