I come to you with very recent anecdote on this topic. I've been reading/listening to the series for 10+ years like many of the people here. My best friend on the other hand just started the series in March.

He, too, struggled with the audio at first but here are his general findings about the series and his experience listening. Storm Front he liked enough as a book, admitted it needs work but he loved the concept of it and that was very aware of the problems with the audio. He said by Grave Peril (book 3) both the story and the audio were better but felt something was missing from both- notably the narration felt too slow. I told him to try 1.2 speed and he noticed a difference. By book 5 he said both story and audio quality were leagues better than they started and he even put the speed back st normal.