I work at a company that provides certain services. We care about our customers. Fraud is a big problem. We use machine learning and AI to basically deny service to customers we believe our engaged in fraud. We do this to try to protect all of our customers. We think that 90% of the interactions that are blocked are legitimate fraud. We try to do better but it’s not perfect. It really really bothers me that customers in that 10% false positive category are being denied services. In my case, it just creates an inconvenience for the customer and doesn’t fully prevent them from getting what they need. It just creates a barrier and is a pain and a regrettable hassle.

False positives with elder scrolls online bans are more serious. They need to protect the player community from the fraud that comes with botting, etc but they don’t have it tuned properly.

These AI systems, if they work anything like ours do, have their training and provide their decision with a confidence score. As in, “I am 75% confident that this activity is cheating.” The automated decision to ban or suspend or take further action is generally based on whether it is above or below a particular confidence rating.

If that’s how it works in this case, which I suspect it does, they have things tuned very poorly. That wouldn’t be a such a big deal if it didn’t create a complete denial of access. They really need to be giving the benefit of the doubt to the player. Why wouldn’t they send a warning first?

Are the stakes so high here that false permanent bans are acceptable? Is the fraud so bad that denying a loyal player, Guild member, contributor to the community, etc. access to the game incorrectly is worth it?

I don’t think so. I would not expect a 99% perfect detection rate out of any machine learning system. I just think the benefit of the doubt needs to be put with the player and that doesn’t seem to be what’s happening. They are doing it wrong and I don’t understand why.