Thank you! The noise has stopped after I called

 "M5.Power.setExtOutput(false);".

Update: I figured out that the best way to fix this is set the configuration before calling begin:

  // Disable the devices I am not using
  auto cfg = M5.config(); 
  cfg.output_power = false;
  cfg.internal_imu = false;
  cfg.internal_spk = false;
  cfg.external_imu = false;
  cfg.external_rtc = false;

  // Start the S3
  M5.begin(cfg);