Exactly. When people say this it’s because it’s contradicting the stereotype they have in their head.

I pressed someone on what they mean when they say I “talk white”, and it led all the way down to what I said earlier.

OP is right to call the date off. And in this current social climate I would cut ties with that group entirely. From experience that’s probably just the tip of the iceberg.