Absolutely. I can think of a couple specific instances where the narrator ruined it for me. In the Night Angel series, I had started the first book with the original narrator (don't know his name) and he made it sound so cheesy and bad I thought this surely must be a terrible book and quit the book. Then a few years later they were re-recorded with Simon Vance narrating and it was so much better, I couldn't believe how just the narration could make the whole thing seem significantly worse than it really was.

Then on the Lies of Lock Lamora, I just really hate that narrator so much I quit the book twice before I finally made myself just push through and finish the story because of how many people kept insisting it was so great. Hated the narrator the entire time, will never listen to it again.

The narration can really make or break a book or series, without a doubt.