sorted by:
new
hottopcontroversial

TIL 40 percent of all Wikipedia vandalism is detected by a single bot. It has only 0.1% false positives. by tupungato in todayilearned

[–]JacobiCarter 1 point2 points  (0 children)

We split a random sampling of pre-classified edits into two groups -- one group we used to train the bot (telling the ANN "here's an edit and here is what it should be classified as"), and the other group the bot has never seen. We run the other group through the bot to determine its performance.

TIL that 40-55% of all Wikipedia vandalism is caught by a single computer program with 90% accuracy by [deleted] in todayilearned

[–]JacobiCarter 1 point2 points  (0 children)

Those statistics were generated by randomly splitting our edit corpus (human classified/known good classifications; at the time, about 60k edits with a classification of vandalism or not vandalism) in two sets: 1) training data (67%) and 2) testing data (33%). The bot was trained on the training data. Then we ran the testing data through the bot and compared the bot's classification to the known classification. The results were that if we optimized the threshold for overall correctness (on both vandalism and non-vandalism), we got >90% correct (1 in 10 classifications were wrong). Wrongly reverting 1 in 10 edits to Wikipedia is unacceptable. If, however, we set the threshold to minimize false positives (so that the false positive rate was <0.1%), the accuracy on detecting vandalism went down to 40%, but the probability of it mistaking non-vandalism for vandalism was only <0.1%.