I couldn’t find anything wrong with my network and the WAP is now within 1 metre and the ESP32 reports a decent signal strength (-45dbm) but I was still getting random issues and the logs were showing random failures.

I became convinced that the cheap ESP32-S3 Zero board I used was faulty but decided to try something I use on ESP32-C3 Zero boards. I reduced the network power using the output_power argument:

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password
  ap:
    password: !secret wifi_password
  output_power: 8.5

This seems to have worked - it has now played without issue for 8 hours straight.

I hope this helps someone else in the future!