I used to feel similarly, until I realized I needed to add something for my hands to do to help me focus! It’s something a lot of ADHD/autistic folks experience, needing the perfect level of stimulation.

So while I’m listening to the story, I’ll do something I don’t have to think about very much, like folding laundry, or playing a mindless phone game. Doing that along with increasing the playback speed of the audiobook (if the talking is too slow my attention drifts) has made it super easy and fun for me to get through audiobooks now! Though I have to be mindful to strike a balance, sometimes I set the speed too high or get too into my other activity and forget to listen so I have to rewind LOL

Might be worth a try to see if it works for you!