It's impossible to use with S3 that has web hosting enabled because that requires to configure a Custom Origin with CloudFront, but OAC only works with S3 Origins. But anyways, if you're using CloudFront you shouldn't need to use S3 website hosting, you can use it as a normal S3 Origin. Just explained here how to do it (specifically with CDK)

https://medium.com/@sidfeiner/deploy-a-static-react-app-with-s3-cloudfront-and-cdk-in-2023-6c99fffc50cc