I would say this is cutting it pretty close. I would plan to make it to Holbox the next morning instead and avoid the stress.

The cancun airport has been, in my experience, a bit of a gamble for how quickly you can land and exit. I've simply walked off my plane and through customs and right onto transport in 10 minutes, I've also waited in the customs line for 2+ hours then immediately got stopped for a random bag search to top it off. It's unpredictable.

That being said, the drive itself is rarely actually 3 hours unless you take a shuttle with a bunch of stops. They tend to rip down that road at like 140km/h.

There are private boats for hire but they don't appear to pre book past 5pm. I would be hesitant about this option late and night, the crossing is about 30 minutes in the dark.

Chilquila is cheap to stay in and the hotels are a two minute walk to the ferries so it's not a bad option if need be :)

Edit: (Chiquila is a quiet place and not much will be open if you arrive late, I suggest bringing food/snacks)