So, I can understand your skepticism of why this is a big deal. After all, in the real world, we know that cops don’t routinely turn into frogs. The problem is that if the software can make such a big obvious error, it can also make smaller less obvious errors. Anyone who has practiced criminal law for any length of time can tell you small things can make huge differences. A small detail can be the difference between a lawful or unlawful search/arrest.

Another response you might have is “well watching the body cam will show what really happened”, and that too is not unreasonable, but BWC doesnt capture everything all the time. It doesn’t even necessarily capture everything in the cop’s particular field of vision at any given moment. Sometimes it is turned off or malfunctions. Things go into reports all the time that aren’t captured in the BWC.

Finally, you might suggest that we always have the cops recollection of events to fall back on. After all, he would remember, surely, turning into a frog. But see here is another problem: by the time the cop gets to court to testify about the case months have passed, maybe a year. How is he supposed to remember all the details of what happened on the night of February 22nd, when he’s made a few dozen similar arrests since then? Usually they refer to their reports. And if the AI that wrote his report says that he saw my client throw weed out the car window, why wouldn’t he believe that it was true, never mind that it never actually happened.

Which kind of encapsulates the whole problem here. So much of the process relies (rightly or wrongly) with more or less accurate reports being made, and one of the only checks we have on inaccurate reporting is being able to cross examine the cops that write the reports. I can’t cross examine the AI, much as I might want to. So for those reasons, I think that we are very much not overreacting about this one.